How to test your AI agent's context against live data before it ships

The analytics agent passed every demo. It answered "active customers by segment" in four seconds, the SQL looked right, and the pilot group liked it.

Then the model risk review sampled 20 answers, and three disagreed with the BI report.

On two of those, the agent had resolved "customer" to the CRM contact table, which counts people, while finance counts billing accounts. The AI team had no way to see that before the review did.

Give the agent its definitions, then prove they hold

Last week's post, Your AI Initiative Has a Metadata Problem, made the case for handing an agent governed definitions before it starts guessing from raw schema. That fixes where the agent's context comes from. It doesn't yet tell you whether the context returns the right answer.

So this post walks through testing an agent's context the way you'd test code: a question set, expected answers, checks on what each concept resolves to, and evidence for every result, run against live data before the agent ships.

Write the questions your users will ask

Start with the business questions the agent will get. The BI reports your users already rely on are the best source, because every figure in them is a question someone asked and an answer the business accepted.

Pick 30 to 50 questions across the metrics the agent will serve. Include the ambiguous ones on purpose: "customer," "revenue," "active," and any term two departments use differently. Those are where agents go wrong.

For each question, record the approved reference answer, taken from the BI report or confirmed by the metric's owner. That pairing is your test suite.

Add a few questions the agent should decline for some users: a regional revenue cut that a sales manager in another region isn't entitled to, for example. Run those with each user's personal access token. A grounding test that only checks correct answers misses the case where the agent returns a correct number to the wrong person.

Run the questions against compiled context

The questions need to run against the same context the agent will read in production. In MetaKarta, that context comes from governed semantic models bound to an ontology through Ontology Modeling, so each business term arrives attached to the metric and the physical assets that implement it.

Context Sandbox is the testing environment for that context. It runs representative business questions against the governed context through MetaKarta MCP, the same server production agents connect to, and verifies each answer against live data.

Testing through the production interface matters. A test run against a copy of the context, or against a different prompt, tells you how a different agent would behave.

Compare every answer with the approved one

Answer evaluation compares each output with the expected result or the approved reference answer. The first run produces a scorecard: questions that match, questions that miss, and by how much.

The misses are the useful part. Each one is a definition, a binding, or a missing concept that would have reached users if the agent had shipped.

Sort the misses by cause before fixing anything. A cluster of misses on one term usually points to one binding, and a scattered set across many terms usually points to definitions the business never agreed on in the first place.

Resist fixing misses in the system prompt. A prompt fix repairs one question for one agent. A fix in the governed definition repairs every agent and every dashboard that reads it.

Check what each concept resolves to

A correct answer can still rest on a wrong binding, so the next check looks underneath the answers.

Binding validation confirms that each ontology concept resolves to the correct lineage, tables, columns, and metrics. This is where the "customer" problem from the opening shows up directly: the concept resolves to the CRM contact table, and the governed definition says billing account.

Fix the binding once, in the ontology, and every question that touches "customer" picks up the correction on the next run.

Inspect the grounding behind each result

Grounding inspection shows which definitions, relationships, and physical assets supported each result. For a reviewer, that's the evidence they asked for: this answer used this definition of active customer, joined these tables, and read these columns.

Lineage then traces from the governed definition back through every transformation to the source columns, so the chain runs from the answer to the data.

Test the claim yourself, on your data, with the grounding visible.

Ship the agent, then keep the test suite

When the scorecard is clean and the bindings validate, the agent can go live through the path it uses: MetaKarta MCP for external agents, with per-user access tokens, or a governed model deployed as a Snowflake Semantic View for Cortex Analyst for example.

Keep the question set. Definitions change, columns get renamed, and new terms get added. Impact analysis flags a schema change that reaches a governed definition before it deploys, and rerunning the suite after any definition change catches regressions before users do.

Where tested grounding pays off

An agent's context can be tested the way code is tested: real questions, approved answers, binding checks, and visible evidence, all against live data and before release. The review that would have found three bad answers in a sample of 20 instead reads a scorecard that was clean before launch.

MetaKarta builds that into the Metadata Management Platform through Semantic Hub, Context Sandbox, Ontology Modeling, and MetaKarta MCP, on the same shared metadata repository as Data Lineage, Data Catalog, and Data Governance.

Get in touch to learn more about testing your agent's context against your own data before it ships.