Field note /
Expected answers partly exposed in a preregistered benchmark
The Semantic Site Evaluator preregistered a synthetic calibration set (its protocol was fixed in writing before any run), then found that missing source text and policy wording could give away the expected answer.
How the finding changed
Belief
A fixed answer key, a stated denominator and a preregistered protocol would make the next semantic calibration result meaningful.
Evidence
A ten-case version 2 contract, a runner that denies access by default, and an independent-review packet were merged. A separate AI review reportedly agreed with the proposed labels. No named human froze the answer key, and no calibration run of Jev (TypeSafe's model-advice service) took place.
Contradiction
The expected answers could partly be guessed from missing-source patterns and from close paraphrases between policy and evidence. Preregistration protected the scoring protocol but did not guarantee that the task tested hard evidence judgments.
Correction
Treat version 2 as smoke-test material only. A version 3 design and a proposed 24-case challenge set were then merged. They separate consistency with a stated policy from checks against primary records that the provider can see. The second still needs a reviewed evidence schema and a rule for what data may leave the system.
Current claim boundary
As of 12 October, version 3 is a merged design and a proposed synthetic set. It has no human-frozen answer key and no working primary-record check. A separate policy-boundary revision and an AI-review register have since merged, but the 24-case human answer key is not frozen. No Jev calibration, field-accuracy result or release gate exists.
Evidence
- Version 2 preregistration, runner and review packet (private repository)Private
- Version 3 challenge-set design and proposed set, PRs #4 and #5 (private repository)Private
- Policy-boundary correction, PR #9 (private repository)Private