From measuring semantic disagreement to measuring decision correctness against an independently frozen oracle.
A semantic system behaving differently from deterministic routing is not evidence that it is better. This project exists to answer the harder question: when the two disagree, which decision is actually more correct? That requires independent ground truth, blinding, explicit adjudication, lossless evidence, uncertainty-aware statistics, and a publication policy that remains valid even when the semantic candidate does not improve the control.
Routing Semantic v1 is a completed, P3-verified 120-case comparative study. The oracle was independently adjudicated and frozen before inference; P1 verified the evidence path; P2 ran all 120 cases with the oracle absent and joined only afterward; and P3 published a sanitized result after explicit privacy/integrity verification. In the frozen cohort, the semantic candidate was more accurate on task class and interaction required, while semantic-risk error was approximately unchanged. Routing Semantic v2 is preserving the same evidence-first discipline during qualification: after several fail-closed integration corrections, Benchmark 0.6.3 passed the full live IA-1 through IA-11 qualification with sanitized proof of strict schema transport. Before any real v2 label was produced, a 120-case readiness review exposed a schema-scale representation problem; Benchmark 0.6.4 reduced the equivalent model-facing schema from 225,971 to 3,310 serialized characters by grouping identical decision shapes. Because the representation changed, the earlier pass no longer authorizes the real cohort. Fresh 0.6.4 qualification is required before real routing-semantic-v2 A/B/C. Separately, a fresh 30-pair Agentic-Jev SWE-manager v1 study completed with 15/30 correct in both control and treatment; its request audit found 0/20 successful Jev calls met the frozen context-completeness check. The corrective v2 manager treatment moves authoritative context construction into deterministic host code, has passed live qualification on an already-observed task, and is preregistered for a fresh effectiveness cohort that remains unrun.
A completed comparative-study and evidence-engineering pipeline that evolves the question from semantic disagreement to independently measured correctness: frozen cases, blinded oracle creation, staged preflight/full-study execution, seam-specific correctness and calibration, request-level accounting, explicit methodological corrections, and a fail-closed public projection verified before publication.
The machinery follows from a stricter research question; it is not the point of the study by itself.
Adjudicators see only the frozen case evidence and rubric, never treatment outputs.
The v1 oracle is preserved as executed. Private reasoning summaries later showed explanatory evidence existed upstream, but the authoritative pass and original resolution workflow had reduced it to labels.
This prevents an application threshold or fallback rule from being mistaken for model quality.
Truth is first joined after inference; correctness and request-level overhead are then reported under frozen sample and publication rules.
A changed decision is only a candidate effect. Whether it is beneficial, harmful, or redundant is measured against independent ground truth.
Ground-truth adjudicators cannot see deterministic or semantic outputs, probabilities, comparison results, or hidden construction tags.
Raw semantic quality is measured separately from confidence thresholds, fallbacks, lifecycle rules, and the final applied decision.
A genuine three-way conflict can remain explicitly unresolved; the study does not invent a label to preserve sample size.
The authoritative v1 pass omitted structured justifications even though private reasoning summaries survived upstream. The frozen cohort is not rewritten; the corrective response is a versioned future contract with provider-neutral structured justification and end-to-end persistence checks.
A valid private oracle was not automatically safe to publish. P3 uses a deliberately redacted public projection and an independent verifier for allowlist, integrity, cohort identity, and privacy boundaries.
A runtime lock is only reproducible when it binds the result-affecting executable surface and rechecks it before qualification or execution. V2 hardening happened before new treatment outcomes, so it changes the readiness boundary rather than the result.
The study succeeds when its evidence is defensible and reproducible, even if the semantic candidate is equal to or worse than deterministic control.
The completed study measures deterministic and semantic routing against independently frozen ground truth, then publishes a separately verified public projection. The result is heterogeneous by seam: semantic-candidate accuracy was higher for task class and interaction required, while semantic-risk ordinal error was approximately unchanged.
Frozen cases
120
public-safe shared corpus
Decision seams
3
task class · interaction required · semantic risk
Minimum eligible / seam
100
preregistered publication threshold
Paired bootstrap
10,000
deterministic resamples · 95% confidence
Oracle path
Frozen + validated
qualified DeepSeek adjudication → dispute-only C → recorded human resolution
Oracle API baseline
$0.07
36 DeepSeek-4.1-Flash requests · 726,263 tokens · through P0B freeze
Decision provenance
Corrected diagnosis
authoritative structured justification omitted · private reasoning summaries retained upstream
Task class
81.67% vs 52.50%
+29.17 pp · 95% +20.0 to +38.33
Interaction required
91.67% vs 71.67%
+20.0 pp · 95% +10.83 to +29.17
Semantic risk
0.28975 vs 0.29167 MAE
paired interval -0.101917 to +0.092917 · spans zero
Provider requests
120 / 120 successful
91,445 input + 10,598 output = 102,043 provider tokens
Request latency
p50 0.1468 s
p90 0.1865 s · p95 0.2206 s
Calibration
Brier + ECE / log loss
interaction 0.07741 Brier · 0.14942 ECE; task 0.27346 Brier · 1.80242 log loss; risk 0.32484 Brier · 0.53531 log loss
Decision changes
30/6 · 38/3
interaction beneficial/harmful · task beneficial/harmful; task also had 10 changed-but-both-wrong cases
P2 full cohort
COMPLETE
120 cases · 360 observations · 120 requests · 0 exclusions
P3 publication
VERIFIED + PUBLISHED
public oracle projection · integrity + privacy checks passed
| Area | Current state | Assessment | Next action |
|---|---|---|---|
| Research question | Answered for v1 cohort | Difference was scored against independently frozen ground truth rather than treated as improvement by itself | Replicate/version future studies rather than generalize beyond the frozen cohort |
| Ground truth | Frozen, validated, and joined post-inference | Independent and blinded before treatment inference | Keep v1 immutable; any future adjudication change belongs to a new versioned study |
| Oracle baseline cost/usage | Recorded | $0.07 · 36 requests · 726,263 tokens via Codex harness -> codex-lb -> DeepSeek-4.1-Flash | Keep this baseline separate from P1/P2 Jev treatment accounting |
| Adjudication | P0A/P0B complete | DeepSeek-Flash cohort completed A/B/C plus recorded human resolution | Do not relabel the frozen oracle; preserve the completed cohort as baseline ground truth |
| Decision provenance | Diagnosis corrected | Authoritative A/B/C pass omitted structured justification; private Inspect reasoning summaries survived upstream but were unavailable to original human resolution | v2 requires provider-neutral structured justification plus round-trip persistence and explicit session-usage scope |
| Model vs policy | Separated | Raw semantic correctness can be measured independently of thresholds/fallback | Report both layers without conflating them |
| P1 instrumentation | Passed | Live evidence path verified on 8 development-only cases with 0 exclusions | Retain as the preflight pattern for future cohorts |
| P2 outcome | Complete | Higher candidate accuracy on task class and interaction required; semantic-risk MAE approximately unchanged | Keep reporting seam-specific; do not collapse into a universal winner |
| P2 provider evidence | Complete | 120/120 requests succeeded; 102,043 provider tokens; p50/p90/p95 request duration 0.1468/0.1865/0.2206 s | Keep request-level efficiency separate from incomplete provider-cost evidence |
| Calibration edge case | Corrected + disclosed | One 0.99-mass vector in each multiclass seam; raw evidence preserved and derived calibration copy normalized only | Move probability-mass validation into future preflight/reporting invariants |
| P3 publication | Verified + published | Private-oracle disclosure path was replaced before release by a redacted public projection with independent verification | Keep private and public evidence contracts separate |
| Public claims | Seam-specific v1 findings published | No universal Jev superiority, downstream quality causality, cost advantage, or material semantic-risk improvement supported | Require replication/new version for broader claims |
120 frozen cases · 3 routing seams · 10,000 paired-bootstrap resamples
The preregistered study specification freezes the corpus target, three decision seams, sample thresholds, exclusion policy, oracle policy, metrics, and deterministic statistical policy before live comparative results are available.
Methodology and study-readiness evidence only. This does not establish that TypeSafe/Jev improves correctness, latency, cost, or any other outcome.
P0A/P0B complete · DeepSeek-Flash adjudicator cohort · oracle frozen + validated
After IA-1 through IA-8 qualification, independent A/B passes labeled the exact blinded authoring view, C received only disputed seams, genuine three-way conflicts went through recorded human resolution, and the resulting oracle was frozen and validated against the 120-case corpus.
This establishes the independent ground-truth baseline later used for scoring. The oracle was frozen before treatment inference and joined only after the full P2 run completed.
$0.07 USD · 36 requests · 726,263 tokens
Operator-recorded provider/API accounting for initial qualification and adjudication through P0B oracle freeze. Execution path: OpenAI Codex harness -> codex-lb load balancer -> DeepSeek-4.1-Flash API; study tooling recorded the routed alias openai-api/codex-lb/deepseek-flash.
Oracle-baseline construction only. Excludes P1 TypeSafe/Jev smoke, P2 comparative inference, and P3 publication. Raw billing/provider traffic is not committed, so these figures are not used to infer per-token pricing or comparative efficiency.
Authoritative labels retained · structured justifications omitted · private reasoning summaries retained upstream
Post-freeze evidence-chain review first identified that the labels-only authoritative pass omitted structured per-decision justification. Follow-up audit of retained private Inspect logs found contemporaneous provider reasoning-summary events, establishing that explanatory evidence existed upstream but was never promoted into the pass or original human-resolution workflow.
Methodology and evidence-completeness finding only. The private summaries are supplementary execution evidence, not hidden chain-of-thought, were not available to the original resolver, and are not used to rewrite the frozen v1 oracle.
120 cases · 360 observations · 120 requests · 0 exclusions
All frozen cases ran with the oracle absent from inference. The frozen oracle was joined only after inference completed. Seam-specific correctness, calibration, disagreement, reliability, latency, and token evidence were then reported under the preregistered policy.
Bounded routing study. Task-class and interaction-required accuracy were higher for the semantic candidate in this cohort; semantic-risk MAE was approximately unchanged and its paired interval spans zero. No universal-superiority, downstream software-quality, or treatment-cost conclusion follows.
Interaction Brier 0.07741 · ECE 0.14942 · task-class Brier 0.27346 · risk Brier 0.32484
Calibration was computed from the persisted semantic probability evidence after the post-inference oracle join. One probability vector in each multiclass seam summed to 0.99; raw evidence was preserved and only the derived calibration copy was normalized.
Calibration evidence for the frozen routing-semantic-v1 cohort. Task-class log loss was 1.80242 and semantic-risk log loss was 0.53531. These metrics do not support a universal model-quality conclusion.
120/120 success · 102,043 tokens · p50 0.1468 s · p95 0.2206 s
Request-level accounting counted the single batched semantic provider request once per case rather than once per decision seam. The run recorded 91,445 input tokens, 10,598 output tokens, and 102,043 provider tokens total; latency was p50 0.1468 s, p90 0.1865 s, and p95 0.2206 s.
Provider request reliability, token, and request-duration evidence for P2 only. Provider cost evidence is incomplete, so these measurements do not support a treatment-cost conclusion.
PASS · exact allowlist · manifest integrity · privacy contract
The publication verifier checked the exact file allowlist, SHA manifest, 120-case/360-observation cohort identity, redacted public oracle projection, privacy flags, and absence of private A/B/C votes, human-resolution rationale/identity, and private reasoning summaries.
Publication-integrity evidence. The public result is a deliberate projection of the valid private evidence bundle rather than a verbatim copy of the private frozen oracle.
PASS · 8 cases · 8 requests · 24 observations · 0 exclusions
A deterministic development-only subset of the frozen corpus exercised the live TypeSafe/Jev path. Verification checked one request per observed case, three seam observations per case, unique request IDs, persisted semantic probabilities/distributions, explicit failures, oracle absence during inference, privacy-safe persistence, single-count request accounting, and frozen routing/v2 plus routing-state/v2 identities.
Instrumentation and evidence-integrity evidence only. The smoke is deliberately below the preregistered study threshold and does not support any TypeSafe/Jev comparative-effectiveness conclusion.
PASS · IA-1 through IA-11 · 15 sanitized ingress requests
An archived Benchmark 0.6.3 retry completed the strengthened qualification. Sanitized loopback capture verified structured-output requests reached Codex-LB ingress with json_schema, strict=true, and schema hashes matching persisted primary/C artifacts; the qualification manifest reported qualified=true and all eleven gates passed.
Qualification evidence for the 0.6.3 schema implementation only. It does not authorize the later 0.6.4 grouped schema and is not a real-cohort oracle result.
225,971 → 3,310 serialized chars · 98.54% reduction · 120/120 IDs preserved
Before any real label, deterministic reconstruction compared the per-case anyOf schema with an eligibility-grouped representation. All 120 frozen cases share the same three decision seams, so grouping removes repeated record shapes while retaining the exact allowed case-ID set and unchanged deterministic post-generation validation.
Representation and readiness evidence only. It does not establish that the larger schema would have failed at a provider, and it deliberately invalidates the 0.6.3 qualification for 0.6.4 authorization.