Skip to content
CodelessOps

Grounded · The checklist

What a RAG agent needs.

A properly built RAG agent needs all of this. Ten areas, seventy-two lines, from parsing the documents to running the thing in production. The demo version skips most of them; the one a finance team can rely on cannot. Under each area is where Grounded stands today, and where that is written up.

The ten areas

Each card is one area a well-built agent has to cover. The bar is where Grounded stands on it today, checked on 2026-10-30. Click through for the lines.

01 · Ingestion

Overview

Ingestion

Everything downstream depends on the structure surviving this stage. A finance workbook flattened to prose has lost the row label, the column header and the sheet name, and retrieval cannot put them back.

Grounded today

  • builtKeep document structure: tables as tables, workbooks as grids, an A1 reference on every cellwrite-up
  • builtStore two texts per chunk: an enriched one to embed, a verbatim one to cite fromwrite-up
  • builtSkip unchanged files on re-upload, because every ingest costs embeddings and an LLM passwrite-up
  • builtFence the parser: size and pixel limits, process isolation, OCR only when a page needs itwrite-up
  • designedA queue that recovers when the process dies mid-document · designed, not builtwrite-up
  • not builtRead charts and images · by choicewrite-up

02 · Retrieval

Overview

Retrieval

Hybrid search, a reranker that is proved to be on, and scope applied inside the search functions rather than filtered afterwards.

Grounded today

  • builtHybrid search, vector plus keyword, fused by rankwrite-up
  • builtA reranker, and a measurement that proves it is onwrite-up
  • builtTenant and folder scope applied inside the search functionswrite-up
  • builtDocument authority: drafts, legacy copies and superseded versions ranked down, never hiddenwrite-up
  • builtA retrieval-only eval (hit@k, MRR) that runs in seconds after any change to chunking, embeddings or rerankwrite-up
  • partialA written account of how one question becomes several searches · the agent decides, up to 25 rounds; the pattern is in the traces, not yet on a page

03 · Answering and grounding

Overview

Answering and grounding

Where the promise is kept or broken: no figure without the cell or page it came from, arithmetic done by code, and a refusal when the documents cannot support the answer.

Grounded today

  • builtCitations that resolve to a cell or a page with a bounding box, not to a documentwrite-up
  • builtCitation parity: the model can only cite what retrieval returnedwrite-up
  • builtArithmetic done by code, with the inputs recordedwrite-up
  • builtRefusal enforced after the model, in code, not requested in the promptwrite-up
  • builtAbstain when the corpus does not hold the answer, and test for it · every golden suite carries refusal questionswrite-up
  • builtSources carry a trust tier the reader can see: documents, analyst notes, webwrite-up
  • builtPer-figure verification shown to the user, not just loggedwrite-up
  • builtLoop limits: tool rounds, sub-agent depth, context compactionwrite-up
  • not builtA token budget per answer and a stop on runaway loops

04 · Guardrails

Overview

Guardrails

The guards that sit after the model and around the tools, so a wrong figure, a leaked name or a planted instruction cannot reach the user.

Grounded today

  • builtOutput guard on figures: the numeral checkwrite-up
  • builtPII: reversible surrogates for people, hard redaction for identifiers, and an invariant that numbers are never touchedwrite-up
  • builtMasking before any event leaves for the trace storewrite-up
  • partialInstructions inside documents do not become tool calls · tested (8 tests); the defence is not yet written upwrite-up
  • builtCode the model wrote cannot do more than the model could: a read-only allow-list on the sandbox bridgewrite-up
  • builtNo per-organisation base URL, so a shared key cannot be pointed at an attacker's serverwrite-up
  • not builtInput scope and jailbreak handling
  • not builtOne page that lists every guard, where it runs and the test that pins it · the material is spread over four pages

05 · Evals

Overview

Evals

Build the eval before the feature. Every stage that can silently degrade to a no-op needs a test from day one, and the results need a date and a place in version control.

Grounded today

  • builtGolden questions per corpus, each with the source a correct answer must citewrite-up
  • builtGrade the answer the user sees, after the guard, not the raw model outputwrite-up
  • builtDeterministic citation assertions beside the LLM judgewrite-up
  • builtA preflight that checks every expected source exists before a token is spentwrite-up
  • builtResults committed as dated files, so the history is in gitwrite-up
  • builtEvals cheap enough to run constantly · after a $50 daywrite-up
  • builtA public benchmark the product sits like any other agent, scored by a machinewrite-up
  • builtA decision on whether evals run in CI · decided: never automaticallywrite-up
  • not builtSwap the chat model and publish the delta on both suites

06 · Observability

Overview

Observability

Every answer should be one click from its trace: what was searched, what came back, what the guard decided, what it cost and how long each step took.

Grounded today

  • builtTyped observations (agent, tool, retriever, guardrail) so a run renders as a graph, not a list of spanswrite-up
  • builtSession id is the conversation, user id on every tracewrite-up
  • builtCost per answer from a model registry, with flat-rate models registered at $0 so tokens still countwrite-up
  • builtLatency per span, read at p95write-up
  • builtRelease and environment on every event, so a regression pins to a deploywrite-up
  • builtA sampled live judge for hallucination, and thumbs-down routed to a human queuewrite-up
  • builtAlerts on traced cost and on judge scores · dailywrite-up
  • builtThe trace store being down never fails a request · a runbook step stops the store and checks the app still answerswrite-up
  • builtThe trace store's secret never reaches the browser · pinned by an end-to-end testwrite-up
  • builtLinks from the product into the traces: the session for a thread, the trace for an answer, the trace on every audit package · 2026-10-30
  • not builtService levels and a dashboard built from real trace data
  • not builtRunbooks: slow, expensive, wrong, stuck, leaking

07 · Audit

Overview

Audit

What a reviewer gets after the fact: one package per answer with the question, the answer, the model, the cost and the trace, kept, and exportable to someone outside the product.

Grounded today

  • builtA package per answer: question, answer, model, tokens, duration, status, trace link · not yet written up
  • builtA read-only mode so packages cannot be deleted through the API · not yet written up
  • builtExport of a package for someone outside the product · not yet written up

08 · Tenancy and security

Overview

Tenancy and security

How one organisation's documents stay invisible to another inside a pooled instance, and how an instance stays closed to the outside.

Grounded today

  • builtScope rules written in one module; retrieval functions take the organisation id; row-level security as the backstopwrite-up
  • builtA drift test that fails if a policy or a function grant changeswrite-up
  • builtSecurity tests by threat, counted and listed · 95write-up
  • builtPer-organisation model keys, envelope-encrypted, model chosen from an allow-listwrite-up
  • builtTokens verified against the identity provider's keys, asymmetric firstwrite-up
  • builtPublic signup off; users created by an administratorwrite-up
  • partialA threat model and a data-flow map of which provider sees what · the PII note covers the model providerwrite-up
  • not builtRetention and deletion · not written

09 · Cost

Overview

Cost

Cost per answer has to be measured and dated, and the levers that move it known: caching, routing by call class, and never re-ingesting casually.

Grounded today

  • builtPrompt caching, with the share of cached input measuredwrite-up
  • builtRoute by call class: the strongest model for chat, a mini-class model for judging, extraction and citation checkswrite-up
  • builtA rule against casual re-ingest, with a file count and a cost estimate before any yes · a standing rulewrite-up
  • builtA sweep of every model and token variable after a provider switch · a rule, after it bitwrite-up
  • not builtPer-tenant metering, a budget and a kill switch

10 · Operations

Overview

Operations

Tests priced in time and money, a trunk that deploys to staging, and production that moves only by pull request.

Grounded today

  • builtTest tiers priced in time and money, with the cheap one impossible to skipwrite-up
  • builtStaging tracks the trunk; production moves only by pull request, with a merge commitwrite-up
  • partialMigrations roll forward, dry-run before every promotion · a rule; the drift scan is not builtwrite-up
  • builtAfter every promotion, confirm a new deployment went active · a rule, after 26 days of silent failurewrite-up
  • builtIncidents written down with the rule each one left behindwrite-up
  • not builtContinuous integration

11 · Where Grounded falls short, in the order I would fix it

  1. 01The guardrails page.
  2. 02The audit trail page.
  3. 03The question-to-answer page: how a question becomes searches, and the loop controls.
  4. 04Service levels, a dashboard and the runbooks.
  5. 05Per-tenant budgets and the kill switch.
  6. 06A threat model with retention and deletion.
  7. 07The model-swap delta.
  8. 08Screenshots, including a trace of one question.

The first three are writing about things that exist. The rest are building.

Hiring?

Need this kind of build on your team?

I take contract roles in finance automation and FP&A systems. Ask me anything about how Grounded works; I'm happy to walk through the parts that broke as well as the parts that worked.