Skip to content
Nate G. / Vancouver, WA · US
all work
MVP / In ProgressTypeScriptNode.jsChromiumLLM Evaluation

Semantic Site Evaluator

An experimental tool that checks a changing website against versioned rules and saved browser evidence, with optional model findings that stay advice only.

SourcePrivate

The source repository and detailed review records are private. The diagrams below explain the architecture but are not public verification receipts.

Case summary

Purpose

Tests that match exact wording break when a site's copy and structure change on purpose. Handing the judgment to an LLM would put release decisions in a system that can be wrong. This project records what must stay true instead of today's exact wording, collects limited browser evidence, checks each rule against the evidence it needs, and leaves release decisions to ordinary code.

System

Semantic Site Evaluator is an experimental prototype that grew out of checking my own portfolio. It loads a declared set of pages in Chromium and gives each versioned rule only the evidence it needs. It records rule-based findings, model findings and operational health separately, and saves a receipt (a record of the result) that can be replayed later. Sending evidence to an outside service is off by default. Each rule is checked on its own, so the whole site never goes into one prompt. A later run on Windows exercised real browser capture, replay and rejection of a modified record. A further run produced one real Jev evaluation of the deployed portfolio after a text-encoding bug was reproduced and fixed. I ran these myself. They do not show that the semantic findings are accurate or that the tool is fit to block a release.

Contribution

An experimental evaluator that keeps versioned rules, browser observation, per-rule evidence, model findings, operational health and release decisions separate.

System diagrams

Each diagram links its Mermaid source and has a static SVG version. They show how control and evidence flow through the system. They are illustrations, not records of real runs.

Approval, per-rule evidence and replay

Limited browser evidence feeds rule-based checks and an optional Jev finding. An unavailable provider is never recorded as a pass, and a modified observation cannot replay.

Approval, per-rule evidence and replay: Limited browser evidence feeds rule-based checks and an optional Jev finding. An unavailable provider is never recorded as a pass, and a modified observation cannot replay.Approval, per-rule evidence and replay: Limited browser evidence feeds rule-based checks and an optional Jev finding. An unavailable provider is never recorded as a pass, and a modified observation cannot replay.
View full-size diagram View full-size diagram Mermaid source

A simplified diagram of the evaluator's documented rule, optional-provider and replay behavior, drawn for this page. The source repository is private. Human sign-off on the expected answers, production use of the claim-support track and a calibrated release threshold are still outstanding.

Supporting architecture views 3 additional views

How the evaluator works

These diagrams explain the system's boundaries without requiring access to private source files. They describe the architecture, not independent verification of a provider's accuracy.

01 / Execution path

From site rules to a replayable receipt

The path from capture to evidence selection to two kinds of decision. This is an architecture model. It does not show that every optional provider path was exercised.

  1. Versioned site rules
  2. Limited Chromium capture
  3. Per-rule evidence selection
  4. Rule-based checks + optional Jev judgment
  5. Finding and operational status
  6. Integrity-checked replayable receipt

02 / Authority model

Three parts, three responsibilities

Browser capture supplies what the tool saw. A model gives a limited opinion. Rule-based software owns integrity, permissions and any release effect.

01 · Browser evidence

  • Page text, headings, links and metadata for the selected pages
  • Coverage and observation identity
  • Observed behavior is never promoted to a rule without review

02 · Model judgment

  • Only the evidence each rule needs
  • Optional provider finding, or abstention
  • A provider failure is recorded as a failure, never as a pass

03 · Rule-based control

  • Schema, approval and evidence integrity
  • Modified records rejected on replay
  • Release stays advice only until calibration is approved

03 / Evaluation contracts

Two kinds of question, two answer sets

The synthetic test set separates whether a page follows its stated policy from whether a factual claim is supported. Neither has a human-approved answer set yet.

Policy consistency

  • Input: selected page text and a declared conditional policy
  • A policy duty that was never triggered passes. A clear violation fails.
  • Ambiguous policy wording goes to a human reviewer

Claim support

  • Input: a claim plus synthetic source-record excerpts
  • Supported, contradicted or not enough evidence
  • The production adapter for source records and the Jev calibration run do not exist yet

Engineering principles

Record what must stay true

Versioned rules describe what must hold while the wording and structure of the site change.

Approve discovered rules before they count

Discovery can propose a rule. A person reviews it before it counts.

The transport path is part of the evidence

The same JSON can decode differently depending on how bytes move between tools, so the byte path matters.

Model findings are advice for release decisions

Provider findings are inputs to rule-based code. They do not give permission to ship.

Problem

  • Exact-copy tests can fail when a site improves, even though its meaning and the evidence behind it have not changed.
  • A model that can judge meaning-level changes should not also decide what ships.
  • If the whole site goes into one prompt, evidence leaks between rules. Each rule should see only the evidence it needs.
  • Provider errors, missing evidence, denied sharing, model disagreement and rule-based failures are different states. They should not collapse into one green or red result.
  • Replaying a result later requires that the saved evidence, rules and observations still match what was originally captured.

Technical Highlights

  • Node/TypeScript command-line tool with real Chromium page capture
  • Versioned site definitions, with a review-and-approve step before discovered pages become rules
  • Per-page rules that each receive only the evidence they require
  • Rule-based, model and operational results kept separate
  • Sending evidence outside is denied by default and needs both operator and site-policy approval
  • Jev (TypeSafe's model-advice service), connected through a provider adapter supplied at run time
  • Versioned evidence bundles that record the rules, evidence, request, model, response and receipt
  • Offline replay that reuses saved evidence without recrawling the site
  • Replay refuses a bundle whose recorded observations no longer match
  • Windows Chrome startup handling that waits for the browser to be ready before continuing
  • Text-encoding fix in the Node/Python bridge after portfolio text with Unicode characters reproduced a Windows decoding failure
  • One operator-reported run on the real portfolio: the rule checks passed, the model check passed, the run itself was healthy, Jev version jev-1.13.0, and the result is advice only
  • A ten-case synthetic calibration set, a runner and a review packet are merged. A named human has not yet signed off the expected answers, and the Jev calibration run has not happened

System Design

  • A site-owned definition lists the pages, content types, which rules apply, exact prohibitions and what evidence may be sent outside.
  • Chromium captures a limited observation of one authorized page.
  • Each rule gets a fresh projection containing only the evidence kinds it needs. Each projection is hashed separately.
  • Rule-based checks run separately from model evaluation and decide exact failures.
  • Running the provider is optional and must be requested. It stops if credentials, evidence, sharing approval or runtime requirements are missing.
  • The receipt records each rule's model finding, operational state, coverage, model, request and response identities, and that the result does not block a release (advice only).
  • Replay checks the evidence and rule identities before re-evaluating and refuses modified bundles.

Rule-Based Control

  • Rule-based code handles schema checks, page and evidence selection, exact prohibitions, bundle integrity, replay refusal and the release effect.
  • Discovery can propose configuration from what it observes, but that does not become an approved rule without review.
  • Sending evidence outside requires both site configuration and operator approval.
  • Model results are advice only. There is no calibrated threshold that could block a release.

Where Models Fit

  • Jev is the first model provider. It judges limited rule criteria over the evidence it is given.
  • A model result is one finding over the supplied evidence. It does not prove the site's factual claims are true.
  • If the provider is unavailable, fails, returns malformed output or lacks evidence, the result is recorded as an operational state and never as a pass.

Failure and recovery

  • Evidence bundles record the exact rule and evidence identities, so an evaluation can be replayed without another browser crawl.
  • A modified observation makes replay refuse before the provider runs.
  • Captured site evidence and provider credentials may be sensitive, so they are excluded from source control by default.
  • The Node/Python Jev bridge now reads raw bytes and decodes them as UTF-8, so Windows default encoding cannot silently change Unicode text.
  • Live acceptance is limited. The Windows run receipts were reported by me and are not published as raw bundles.
  • The corrected calibration set and the reviewer list are merged, and three AI reviews of the synthetic packet were run. Packet-byte identity and one reviewer date are unverified, and two disputed cases need a human decision. There is no named independent human sign-off, no Jev calibration run and no calibrated release threshold.

Measured evidence

Real-browser capture and replay acceptance

measured

live capture and replay, modified record rejected

Native Windows operator run against the public portfolio after the Chrome startup fix. One live capture was replayed offline and a modified observation was rejected.

Run by me. The raw local run bundles were not archived in a public repository. This covers the browser and replay path only. It says nothing about how well the semantic findings work.

n=12026-10-09source 801b81c0d487

Real-portfolio Jev evaluation

measured

rule checks passed · model check passed · run healthy · advice only

One authorized operator-run browser capture of the deployed portfolio, followed by the official Jev helper. The reported receipt resolved jev-1.13.0 and kept the provider, helper, request and response hashes.

A single run by me. It does not show that the semantic findings are accurate, that the result repeats, that the facts were verified, that the deployed build matches the source, or that the tool can block releases.

n=12026-10-10source 1505e576b367

Evidence & documentation

These references record where the work came from, but the repositories are private. The diagrams above explain the design; they don't replace the primary evidence.

Repository (private)Private

Visitor analytics: page interactions, source category, browser information, and network IP metadata are recorded in private server logs. No analytics cookies are used. Opt out of future engagement beacons in this browser.