Professional profile
Nathan Gallo
Senior AI Platform Engineer
AI agent platforms · Evaluation · Developer productivity
I build AI agent systems and the tests that check them. That includes multi-step agent workflows, evaluation tools, and integrations with MCP (Model Context Protocol) servers.
15+ years of production software engineering, including financial-services platforms supporting $10M+ in daily payment transactions.
Vancouver, WA / Portland Metro · US-based · Open to senior US-based AI / agent engineering opportunities.
Software / systems / agents
Software Engineering
Work Atlas
Recent work on AI coding agents, and the checks I run on what they build.
The Work Atlas collects the projects, studies, failures, and corrections behind that work.
Engineering foundation
15+ years of production software engineering: backend services, APIs, databases, cloud and CI/CD, and legacy modernization. I now apply that to AI agent systems, evaluation, and developer tools.
.NET / C#production backendHow the projects connect
Separate tools that pass work and evidence to each other.
Codebase context
Reading source code and its structure.
Specifications & handoff
Requirements, decisions, and saved baselines.
Running the work
Isolated runs, recovery, review, and acceptance.
AI-assisted checks
Model answers used as advice, with uncertainty stated.
Measuring results
Comparing runs and keeping the evidence.
Each capability group is a way to sort the projects. A project can appear in more than one group.
Numbered arrows show documented relationships between separate projects. Dashed lines mark optional integrations. Select a node to see its connections.
Swipe sideways to see the whole graph.
No project selected
Engineering Questions
Each project started from a specific engineering problem. The links show the design response and the evidence behind it.
Engineering questionProject / workDesign responseInspect
Engineering questionSome companies prohibit third-party MCP servers, but teams still need repository indexing and codebase context.
Design responseOffer DeusData's upstream MCP or my modified local CLI fork, depending on the environment's policy.
Engineering questionRequirements and decisions were getting lost in chat history and needed saved, versioned handoff files.
Project / work
SpecGen-AW
Design responseWrite requirements, decisions, and source baselines down as versioned files before any work runs.
Engineering questionA worker process can finish without the work being complete, reviewed, or accepted.
Project / work
Agent-Workflow
Design responseKeep running, evidence, review, and acceptance as separate steps. Ordinary code decides acceptance. Save enough state to recover and resume.
Engineering questionSome routing decisions benefit from a model's judgment, but policy and the final decision should stay in ordinary code that can be inspected.
Project / work
TypeSafe / Jev
Design responseLimit each model call to a structured question, fall back to the rule-based path when there is no usable answer, and keep application code in charge of the result.
Engineering questionExtra orchestration is only worth keeping if its cost and effect can be measured against a simpler baseline.
Project / work
Benchmark tooling
Design responseRun paired experiments, keep the software produced and the measurements, and publish limitations with the results.
InspectPaired runs and methods
Engineering questionA model-assisted router can disagree with rule-based routing without being more correct. How do you tell a difference from an improvement?
Project / work
Comparative Evaluation
Design responseFreeze the cases and an independent answer key, built without seeing either system's output, before comparing. Two reviewers label each case. A third labels only the disagreements, and people resolve real three-way conflicts. Save the reasons for each decision. Score the model's accuracy separately from the policy applied to it.
Engineering questionHow do you test a site whose content keeps changing, without freezing its wording or letting a language model decide releases?
Project / work
Semantic Site Evaluator
Design responseRecord the site's intent as versioned data, collect browser evidence, give each policy only the evidence it needs, and track operational state. Ordinary code handles integrity, what leaves the system, replay, and release effects.
Decisions & Evidence
What the work taught, and the evidence behind it.
What the work taught
Finished work gets its own acceptance step
When a worker finishes, the requested work may still be incomplete. Running, evidence, review, and acceptance are kept as separate states, with written rules for accepting a result.
Completion gatesMore principles6 additional records
A valid request can still be missing evidence
A well-formed request that was sent successfully can still leave out the evidence the decision needs. In the first manager study, 0/20 successful Jev calls included the full task context, even though the bridge kept every string it sent. How the request is built is now a versioned part of what the study tests, and code builds it.
SWE-manager v2 request policyCombine model answers under defined rules
The version 2 audit found that a complete request can give a useful first answer without improving the overall workflow. Using Noul (a true/false judgment) to rate whether the evidence was sufficient did not tell whether the Choice answer was right, and retrying with the same agent added work without improving the signal. The plan keeps the first answer as evidence and lets code combine Choice, Noul, Score, review, and escalation.
Replay check (pull request 103)Independent verification needs a different method
A Codex-LB shutdown investigation used separate steps: reading the code, reproducing the problem, an independent verifier using a different method, and a remeasurement by the orchestrator. Later upstream review found another wait-time risk and replaced a regression trigger that only correlated with the bug with a deterministic barrier. The upstream pull request is still under review.
Upstream Codex-LB reviewSay exactly what was tested
A passing regression test does not make the whole workflow pass, and a successful run on one system does not cover a different revision. In Codebase Memory pull request 20, the checked-out revision, tested revision, test environment, merge state, worker completion, and final acceptance were each recorded separately.
Codebase Memory pull request 20Filing Harbor keeps private data in the browser
Filing Harbor keeps draft data and source documents in browser memory while the page is open. The server receives only the limited inputs for one calculation. The public routes have no way to save tax drafts.
Filing Harbor case studyEach pipeline touches only its own services on the tested path
Rocket Tooling Designer now publishes through its own services, and the portfolio pipeline owns web. On the path tested, each pipeline touches only its own services. This is not a guarantee about all shared infrastructure.
Rocket Tooling DesignerEvidence in context
TypeSafe / Jev
Jevstructured answersTypeSafe/Jev answers structured questions as advice. Application code still decides policy, what happens next, review, and acceptance.
Inspect evidence limits and integration detailReceipts · qualification evidence
The integration uses TypeSafe’s Choice (picks one option from a list), Noul (judges whether a statement is true), and Score (rates something on an ordered scale). Each call leaves a receipt: source references, question versions, model details, request hashes, and outcome status. Uncertain answers, no-decision outcomes, and fallbacks are recorded.
Application code decides how the advice affects the result. TypeSafe/Jev does not control orchestration, side effects, review, or acceptance, and it can be used without Agent-Workflow.
Checkpoint outside the agent. Skill tests in controlled conditions did not carry over to real use: six real manager decisions made zero Jev calls. Agent-Workflow now has a prototype checkpoint outside the agent. It records the agent’s draft first, sends Jev evidence that leaves out the agent’s verdict, and requires reconciliation when they disagree or the answer is uncertain.
Context. Richer context can make a model’s answer more decisive without making it better at telling cases apart. The independent request now separates tool output from the agent’s own claims, and by default leaves out the agent’s preferred answer and earlier Jev answers.
First manager study. A 30-pair study on SWE-Lancer, a public software-engineering benchmark, scored 15/30 correct with and without TypeSafe (a 0.0 point difference). Comparing each pair, the 95% range was -16.7 to +20.0 points, which means the study cannot tell a gain from a loss. Jev was called on 20 attempts in the TypeSafe setup, but 0/20 successful calls included the full task context. The result is kept as it ran, without repairing it afterward.
Fixing the request and version 2. An audit showed the bridge did not truncate the successful calls. The coding agent had compressed the task and proposal text before sending it. Version 2 builds the request in code. On a fresh 30-pair set, 39/39 successful Jev calls included the full context. Without TypeSafe the score was 16/30, and with TypeSafe it was 15/30. The TypeSafe setup used 27.0% more tokens per task and took 27.46 seconds longer.
Combining answers. Reviewing the exact saved requests showed the first Jev Choice was right on 17/30 tasks, against 16/30 without TypeSafe and 15/30 for the final TypeSafe setup. Nine low-sufficiency cases triggered a second request. None passed the frozen sufficiency threshold, and one Choice changed, from correct to incorrect. The next step keeps the first answer as evidence and combines narrower Choice, Noul, and Score outputs under code-defined rules, without retrying by default.
Limits. A provider call that succeeds does not show the answer was correct. The routing-semantic-v1 study measured accuracy on a frozen 120-case set. Those findings do not show that Jev is better in general or that it improves downstream software quality.
BM5 — one development pair, comparing a direct baseline with a slimmed-down workflow
evaluationbenchmark + evidenceOne retained development pair used approximately 185 seconds less wall time and 235,308 fewer provider tokens with the slimmed Agent-Workflow setup, while scoring 2.5 automated-score points lower.
Limits: one development pair, no human-complete pair, and no measured cost saving.
Inspect saved files and limitationsEvidence for both setups
Paired benchmark runs keep the software produced, along with timing, token use, scoring, and methodology, so a change in process can be compared with a baseline.
This was one development pair. It does not show a general performance effect.
The pair is awaiting human review, and no human-complete pair is recorded. There is no cost data, so fewer tokens cannot be read as a measured cost saving.
TypeSafe qualification ran before both setups, and semantic routing was outside the comparison. The results do not show a TypeSafe effect.
Structured-direct control
Agent-Workflow slimmed
Comparative Evaluation
evaluationindependent correctnessThe study first asked whether TypeSafe/Jev-assisted routing behaves differently, then whether it is more correct, judged against a frozen, independent answer key.
Task class81.67% assisted vs 52.50% rule-based
Interaction required91.67% assisted vs 71.67% rule-based
Semantic riskaverage error 0.28975 vs 0.29167 · interval includes zero
Current limits: v1 is published. V2 passed a full 0.6.3 qualification, but the schema changed in 0.6.4 and must qualify again before real labeling. No real v2 result exists.
Inspect method, corrections, and study filesAnswer key · saved reasons · publication
Early instrumentation showed that TypeSafe/Jev and rule-based routing sometimes behaved differently. That did not say which decision was better.
Earlier questionDoes TypeSafe/Jev behave differently?
Better questionWhen it differs, is it more correct?
NeededAn independent answer key
SoFrozen cases, blinded labeling, and a frozen answer key before any comparison
The answer key is frozen. The study documents call it the oracle, which is the answer key. A group of DeepSeek-Flash reviewers labeled each case independently as A and B on the same blinded view. C labeled only the disputed cases, and real three-way conflicts went to recorded human resolution. The answer key was frozen and validated before any live TypeSafe/Jev comparison.
The study ran in stages. A first run on 8 development-only cases checked the live path. The main run covered all 120 frozen cases without showing the answer key during inference, giving 360 observations from 120 provider requests with 0 exclusions. An independent check verified the redacted public version before it was published.
The results differ by measure. In this study, the TypeSafe/Jev-assisted routing was more accurate on task class and interaction required, and semantic-risk error was about the same at the study’s resolution. All 120 provider requests succeeded. Provider cost data is incomplete.
Model accuracy and operational policy are scored separately. Choice, Noul, and Score outputs are compared with the answer key without counting the current confidence threshold, fallback rule, or non-automatable risk policy as model error.
Intended recordLabels and the reason for each decision
Main v1 passLabels saved; reasons not saved
Next studyReasons saved and checked after a round trip
Missing reasons. The main v1 labeling passes saved labels but not per-decision reasons, so the person resolving conflicts worked from the case, metadata, rubric, and labels. A later audit found the provider’s reasoning summaries in private logs. The resolver did not have them, they are not hidden chain-of-thought, and they are not being added to the frozen answer key. The reasons existed but were never saved to the decision record, so they were not lost completely.
Reporting correction. One probability vector in each multi-class measure summed to 0.99. Raw evidence was kept, only the derived calibration copy was normalized, and the change is disclosed.
Publication correction. Review found the first publisher would have copied the private answer key as it was. The released bundle uses a redacted public version and passed an independent file-list, manifest, and privacy check before release.
Results. Routing Semantic v1 is finished and published. It used 120 frozen cases and 360 decisions from 120 TypeSafe/Jev requests, with none excluded. On task class, the TypeSafe/Jev-assisted routing was right 81.67% of the time against 52.50% for the rule-based control. On interaction required, it was 91.67% against 71.67%. Both differences stayed above zero in a paired 95% interval. For semantic risk, the average error was 0.28975 against 0.29167, and the interval includes zero, so there is no clear difference. All 120 requests succeeded. Cost data from the provider is incomplete, so the study makes no claim about cost, general superiority, or software quality. Version 2 of the study is not finished. Its first full qualification pass worked, but before any real labeling, review found that the schema sent to the model had grown to 225,971 characters because the same record shape repeated for every case. Benchmark 0.6.4 groups the cases and cuts it to 3,310 characters. Because the schema changed, the earlier pass does not cover it and qualification must run again. No real version 2 results exist yet. Separately, a 30-pair SWE-manager study has two versions. Version 1 scored 15/30 correct in both setups, and 0/20 successful Jev calls included the full task context. Version 2 built the request in code and ran a fresh set of tasks: 39/39 successful calls included the full context, the setup without TypeSafe scored 16/30, and the setup with TypeSafe scored 15/30. The TypeSafe setup used 27.0% more tokens per task and took 27.46 seconds longer. A closer look found the first Jev Choice was right on 17/30 tasks before retries. Retries and reconciliation brought the final Jev Choice to 16/30 and the final TypeSafe setup to 15/30. The direction that follows is narrower Jev questions combined by code, with no retries by the same agent. These are separate studies and their results are not pooled.
- Study state
- Complete · redacted public result published
- Full study
- 120 cases · 360 observations · 120 provider requests · 360 answer-key outcomes · 0 exclusions
- Classification
- Task class +29.17 percentage points · interaction required +20.0 percentage points · both paired 95% ranges above zero
- Semantic risk
- Average error 0.28975 assisted vs 0.29167 rule-based · paired interval includes zero
- Requests
- 120/120 success · 91,445 input tokens · 10,598 output · 102,043 total · 95th-percentile time 0.2206 s
- Reasons for decisions
- Not saved in the main v1 pass · reasoning summaries kept only in private logs
- Publication
- Redacted public answer key · exact file list · manifest integrity · privacy checks passed
- Limits
- No claim of general superiority, effect on downstream quality, cost advantage, or material semantic-risk improvement
Evolution / Field Notes
The main changes that led to the current approach. A longer timeline and the field notes are below.
Major turning points 2026
February 2026
Delegating work to Codex from Claude
A Claude-to-Codex wrapper for limited tasks. It explored how to check evidence and the gap between a process exiting and what an agent claims.
invoke-codex-from-claudeArchivedMay 2026
A written job format
A job format that works with any executor separated the task, its scope, its outputs, and where results came from. I tried Copilot and manual packages and a mock executor. The multi-executor runtime was never finished.
agent-jobUnfinished precursorJuly–August 2026
Agent-Workflow: tracked, recoverable runs
Delegation grew into isolated runs, restart history, saved evidence, independent review, and a recorded acceptance step, with a core that runs without a user interface.
Agent-WorkflowContinuing development
Related projects Intake, coordination, and reliability
- ai-reliability-fw
Reliability investigation
- stt-n8n-promptbuilder
Structured intake and prompt delivery · archived
- prompt-forge-console
Operator console for PromptForge · archived
- kanban-prompt-companion
Prompt review and task coordination
Related investigations that the main projects do not depend on.
Later timeline13 recorded milestones · August–October 2026
26–27 Aug
View changeA simpler everyday path
Cut routine workflow steps while keeping each stage of the work separate.
30 Aug
View changeSpecGen records the source version
SpecGen began recording versioned decisions and the exact source baseline as part of the handoff to execution.
September
View changeCodebase Memory as a command-line tool
The modified CLI fork gives restricted environments a local command-line option. It keeps the upstream engine and attribution.
18–19 Sep
View changeTypeSafe adapter and comparison records
The TypeSafe adapter and the shared comparison records were built as separate pieces that can each be inspected.
23 Sep
View changeBM5, a one-pair development run
The slimmed-down worker closeout was compared in one pair, with the software produced kept for inspection. The evidence was mixed.
24–26 Sep
View changeAnswer key frozen, then an 8-case smoke test
A group of DeepSeek-Flash reviewers produced the blinded answer key, and it was frozen. An 8-case development-only TypeSafe/Jev test then checked the setup before the 120-case study: It made 8 provider requests and produced 24 decisions, with 0 exclusions. It also checked that the answer key stayed hidden and that requests were counted.
26–27 Sep
View changeMissing decision reasons found
After the answer key was frozen, review found that the main v1 labeling pass saved labels but not the reason for each decision. A later audit found the model's reasoning summaries in private logs, so the reasons existed but were never saved to the decision record. The frozen answer key is unchanged.
27 Sep
View changePublic result published after a privacy fix
The full 120-case study finished with 360 observations, 120 provider requests, and 0 exclusions. Review found the first publishing script would have copied private answer-key details. A redacted public version passed file-list, manifest, sample, and privacy checks before publication.
27 Sep
View changeOpenSpec considered for planning
Building SpecGen and a versioned shared handoff made the line between planning and execution clear. Reviewing OpenSpec, a planning tool, suggested that planning could use an upstream standard while Agent-Workflow keeps its own execution and evidence formats. This is a direction only. Qualification and migration were still pending at that point.
1 Oct
View changeA check outside the agent
A Jev skill that worked in controlled tests made zero calls across six real manager decisions. Agent-Workflow now has a prototype checkpoint outside the agent: draft first, then independent evidence, then reconciliation when they disagree or the answer is uncertain. The mechanism is merged. Its effect has not been measured.
2 Oct
View changeFirst 30-pair manager study
Thirty paired SWE-Lancer manager tasks scored 15/30 correct with and without TypeSafe: a 0.0 point difference, with a 95% range of -16.7 to +20.0 points across the pairs. Jev was called in 20 attempts in the TypeSafe setup, but none of the calls included the full task context, so request quality became the next problem.
2 Oct
View changeBuilding the request in code
The version 2 manager setup builds each request in code. It reloads the exact task and proposal text from the pinned source, keeps the request identity through dispatch, and rejects unchanged revisions.
10 Oct
View changePrivateUTF-8 encoding bug on Windows
A real portfolio Jev request exposed a Windows bug: UTF-8 text was decoded as CP1252 before the request was built, which tests with ASCII-escaped text missed. The bridge now reads raw bytes and decodes UTF-8 explicitly. One later user-run check of the portfolio passed operationally, with advice only.
How claims changed
Field Notes
Recent public notes. Select one to see how a claim changed.
01 / 07
Accepted run, empty patch, real change
In one accepted OpenSpec task run, the saved patch file was empty, although Git showed a 273-byte source change. The saved record still passed its integrity check, because that check only covers the bytes that were saved.
Agent-Workflow · saved patch evidenceCollector fix merged · clean-machine release acceptance still blocked
Belief
The patch saved with an accepted, reviewed run would match the source change that had been made.
Evidence
While testing two independently accepted OpenSpec tasks (OpenSpec is a planning tool), one accepted run held a zero-byte patch.diff. A separate Git diff of the same change was 273 bytes. The run's integrity check still passed.
Contradiction
The patch collector inherited an empty external-diff setting, so Git exited with an error. The collector treated that failure as an empty patch and saved it. The integrity check correctly confirmed the saved bytes were unchanged, but the bytes were wrong.
Correction
Collecting a patch and saving it are now separate steps. The collector turns off external diff helpers, rejects any Git error, timeout, cancellation or truncated output, and saves a patch only when collection finished cleanly. A genuinely empty diff is still valid. A failed collection saves no patch. Earlier saved records were not rewritten.
Current claim boundary
The fix was merged in Agent-Workflow PR #77, after PRs #66, #73 and #75. Its author reported 47 focused tests, 10 installed-product OpenSpec acceptance tests, a real OpenSpec 1.13.2 smoke run and a 393-passing technical release check. These are reported results, not a new independent CI run or a clean-machine release check. Clean-machine release acceptance is still blocked. Two-task integration needs a fresh combined acceptance, and merging or pushing the work remains a separate decision.
Live Hosted Applications
Applications you can use directly, separate from the engineering tools above.
Grounded
Grounded turns a plain-English analytics question into a structured query plan using an LLM. The app checks the plan against allowlists (lists of permitted tables, columns and operations). It then builds parameterized SQL from approved pieces, runs it, and writes an answer from the returned rows.
The demo asks visitors to verify an email address and answer one short prompt, so anonymous traffic cannot spend the paid model calls.
Rocket Tooling Designer
Rocket Tooling Designer is a browser-based designer for spindle and rammer tooling. You enter a tube size and manufacturing tolerances. It produces an annotated drawing and downloadable review and CAD files. Rocket Tooling Designer (RTS) deploys on its own. The portfolio site forwards /rts/ requests to a separate container that serves this tool.
Filing Harbor
Filing Harbor is a public tax demo that does not keep your data. It is for demonstration only, is not tax advice, and is NOT FOR FILING. Your draft, household names and addresses, dependents, correction history and imported documents stay in browser memory for as long as the page is open. The browser does not use localStorage, sessionStorage or IndexedDB for tax records. The server receives only the inputs needed for each calculation or eligibility check, and fixed code does the tax arithmetic and applies the rules. Reloading, resetting or logging out clears the tax data held in the browser.
Case studyRepositoryPrivate