Field note /
Balancing labels by quota put wrong labels in a draft answer key
The Semantic Site Evaluator's first calibration set used equal numbers of pass, fail and insufficient-evidence cases. Independent AI-agent reviews showed that some cases carried the insufficient-evidence label only to fill the quota. This was found before any accuracy study of Jev (TypeSafe's model-advice service).
How the finding changed
Belief
A balanced mix of pass, fail and insufficient-evidence cases would prevent trivial model behavior and give a defensible answer key for both of the evaluator's checks: one compares page text with a stated policy (the policy check), the other compares claims with primary records (the record check).
Evidence
The first proposal had 24 cases, four of each label in each check. A Claude review that could not see the author's labels agreed with 20 of the 24 labels and challenged four insufficient-evidence cases in the policy check. A second, informed pass found more cases where it was unclear whether a policy applied. Later reviewers disagreed on whether a claim was about an organization or a person, and on whether an outdated, narrow certificate contradicts a broad platform security claim.
Contradiction
The policy check reads the full selected page text against a conditional policy. A duty that the page never triggers does not call for an insufficient-evidence label, and an ambiguous requirement needs a policy decision before it can be scored. Only the record check can legitimately lack enough primary-record support.
Correction
Keep the original proposal, write the policy-check fixes into a separate revision, and use different label sets for the two checks with no quota. Keep author-only labels out of the reviewer packets, and record every disagreement. The majority AI vote is not taken as truth.
Current claim boundary
The revision and a separate record of the AI reviews are merged. Three AI-agent sessions agreed unanimously on 22 of 24 cases and matched the author's proposal on 70 of 72 judgments, across two model families. This is agreement among AI reviewers on synthetic cases. It is not a human answer key, a frozen dataset, a working record check, a Jev calibration run or a release gate. Windows test results were reported by the operator, not by CI.
Evidence
- Policy-boundary correction (private repository)Private
- Merged revision, PR #9 (private repository)Private
- Independent human review template (private repository)Private