Purpose
An LLM is most useful here for working out what the user means. Application code should keep control of the database.
Live.NETStructured OutputsPostgreSQLEvaluation
Grounded
Ask an analytics question in plain English. A large language model (LLM) plans the query, application code builds the SQL, and the answer is written from the returned rows. A benchmark harness scores runs.
The demo asks visitors to verify an email address and answer one short prompt, so anonymous traffic cannot spend the paid model calls.
Case summary
System
Grounded turns a plain-English analytics question into a structured query plan using an LLM. The app checks the plan against allowlists (lists of permitted tables, columns and operations). It then builds parameterized SQL from approved pieces, runs it, and writes an answer from the returned rows.
Contribution
I set where the model's authority ends, the code path that builds and runs the SQL, how answers are grounded and evaluated, and the acceptance criteria. I direct and review the backend and evaluation work against those rules.
Product views
Screenshots of the interface, taken from the portfolio repository. Scroll sideways to see each one. Open an image for full size.
Ask an analytics question
Where the user types an analytics question.
Inspect the structured plan
The structured query plan the model proposed.
Inspect compiled SQL
The SQL view. The compiler builds the SQL from a validated plan.
Inspect the grounded result
The result shown after the database query runs.
Example screenshots already stored in the portfolio repository. They do not confirm the currently deployed build, model accuracy, or the latest evaluation results.
System diagrams
Each diagram links to its Mermaid source (a text format for diagrams) and has a static SVG version. They show how control and evidence move through the system. They illustrate the design and do not record real runs.
Request pipeline: two model calls, SQL built by code
How a live question moves through the system: rejected plans, two separate model requests (plan and answer), and a saved trace (a record of each step).
Adapted from the request-pipeline Mermaid diagram and notes in the Grounded repository, at commit f30bd05e. Details left out do not mean the behavior is absent.
Supporting architecture views 3 additional views
System diagrams
These diagrams show the architecture, who is responsible for what, and where evidence comes from, based on the project record. They describe the implementation and its stated limits. They are not independent test results.
01 / Execution path
The model never writes the SQL that runs
The model works out what the user is asking. Application code validates the plan and builds the database query. The answer comes from the returned rows.
- Natural-language question
- Model proposes a structured query plan
- Check plan against allowlists
- Build parameterized SQL
- Run on PostgreSQL
- Write answer from returned rows
- Score the answer against the rows
02 / Who controls what
Model, application and database each own a different part
The model cannot skip the compiler or run a query itself. The policy rules and the rows the database returns limit the response.
Model · understands the question
- Interpret the user's question
- Propose a structured plan
- Summarize only the query results it is given
Application · rule-based checks
- Allow only approved dimensions and operations
- Build parameterized SQL from approved pieces
- Control query execution and request handling
Database + evaluation · results
- PostgreSQL returns the rows the query produced
- Answer grounding is checked against those rows
- A new accuracy result for the current commit is still pending
03 / Comparison
What the LLM proposes and what code runs
A proposed plan gives no permission to query any table or column. The application either accepts a validated plan or rejects it.
Proposed by the model
- Reading the question
- Structured filters, groupings or measures
- Plain-language summary of the results
Run by code
- Schema and allowlist checks
- SQL text built by the compiler, never by the model
- Values passed as parameters, database access and scoring rules
Problem
- Text-to-SQL, where the model writes the SQL itself, gives the model too much control over the database.
- Generated answers need to be checked against the actual query results.
- Changes to the planner or the answer step need repeatable regression checks.
Technical Highlights
- The LLM planner must return a strictly structured query plan
- The plan is checked against allowlists before any database access
- A fixed compiler builds parameterized SQL, so user values are passed as parameters
- A second safety check runs on the generated SQL
- The answer is written from the result rows
- The planner and answer steps can be swapped for stand-ins in tests
- A test mode runs without API keys and gives repeatable results
- Benchmark scoring and comparison with earlier runs
System Design
- question → query plan → plan check → SQL compiler and safety check → Postgres → answer from rows → evaluation
- The LLM never writes SQL.
- Values that come from the user reach Postgres only as parameters.
Rule-Based Control
- Checking the plan against allowlists
- Building the SQL and passing values as parameters
- Running the query on the database
- Rules for checking whether answers match the returned rows, and for catching regressions
Where Models Fit
- Turn the user's question into a constrained query plan.
- Write a plain-language answer from rows the database has already returned.
Failure and recovery
- The planner and answer steps can be replaced with stand-ins in tests.
- The pipeline can run in a repeatable test mode.
- Evaluation compares new runs with earlier benchmark results.
Measured evidence
New evaluation run on the current commit
not yet publishedRerun the evaluation matrix on the current code. Record the provider, model and prompt, case count, latency, usage, failure types and limitations.
Older evaluation results are historical and are not shown here as current performance.
sample size: not measuredsource 389a522ae1ef
method and source