A practical loop for keeping AI agent evals fresh: capture production traces with outcome labels, curate the failures worth testing, convert them into property-based eval cases, and verify fixes against both evals and production failure rates.
A practical framework for deciding when off-the-shelf models have reached their limit—and when your data, economics, and constraints justify building your own.
Retrieve-for-Train suggests a practical design shift: learn repeatable search behavior offline, but keep live facts and permissions online.
Persistent memory is becoming an infrastructure feature. The harder problem is deciding what an agent may remember, trust, update, and forget.
Most AI agent evals die the way most unit tests die: they were written once, against yesterday's failures, and never updated. Production moves on. The eval suite doesn't. Six weeks later, you're green on evals and red in production, and nobody can explain the gap.
I run into this pattern in enterprise platform work constantly. The fix isn't more evals. It's a loop: production traces in, curated failures out, new eval cases committed like regression tests, fixes verified against them. This isn't another general reliability framework — it's the specific operating loop that keeps evals fresh as production shifts. Here's a practical way to build it.
A typical agent eval starts as a heroic effort. Someone collects 50 golden examples, writes a scorer, gets CI green. Then:
1. The product changes. New tools, new prompts, new user segments. 2. The failure modes change. The old evals still pass because they test the old world. 3. Nobody owns adding new cases. Production incidents get a postmortem; evals get nothing.
The result is a familiar systems problem: an open loop. Observability without feedback. If you've done SRE work, you know the antidote — every incident produces a regression test. Agents need the same discipline, but the "incident" is often a quiet failure: a wrong tool call, a plausible-but-unhelpful answer, a policy near-miss that a human corrected.
I think of it as four stages, each with an owner:
Stage
Question
Owner
Capture
What happened, with enough context to reproduce it?
Platform / observability
Curate
Which failures are worth turning into tests?
Product + engineering
Convert
Can this failure become a deterministic eval case?
Engineering
Check
Did the fix actually move the metric?
Engineering + product
The key constraint: this is selective. You don't convert every trace. You convert the ones that teach you something durable.
Most agent tracing captures the wrong thing: a wall of spans with no outcome label. A trace without a verdict is just expensive logging.
At minimum, I log these fields per run:
# Structured agent run record — store alongside your trace
run_record = {
"trace_id": "tr_01h9...",
"timestamp": "2026-10-04T12:31:00Z",
"agent_version": "support-agent@1.4.2", # prompt + tools + model pinned
"inputs": {"user_msg": "Where is my invoice for March?"},
"tool_calls": [
{"tool": "search_docs", "args": {"q": "invoice march"}, "latency_ms": 320},
{"tool": "get_billing", "args": {"month": "2026-03"}, "latency_ms": 410},
],
"final_answer": "I couldn't find your March invoice...",
"outcome": {
"verdict": "failure", # success | failure | corrected
"failure_class": "retrieval_miss", # taxonomy below
"human_override": True,
"correction": "Agent searched docs instead of calling get_billing first",
"severity": "medium",
},
}Two things matter here more than the schema:
Pin the agent version. Prompt, tools, model, and policy config are a single deployable unit. If you can't say which version produced the trace, you can't say whether the fix worked. Treat it like a container image tag.
Redact at write time. Traces contain user data. Decide retention and access controls before you have a thousand traces, not after. I keep raw user text for 30 days, structured outcomes for 13 months, and eval cases indefinitely — with PII stripped during conversion.
A minimal failure taxonomy keeps curation sane. Mine usually starts with five buckets: retrieval_miss, wrong_tool, tool_args_invalid, policy_near_miss, unhelpful_correct (factually right, useless in practice). You'll rename them within a month. That's fine.
This is where most teams drown. A busy agent produces thousands of imperfect runs a week. Converting all of them is how you get a 10,000-case eval suite nobody trusts.
I curate weekly, in 30 minutes, with one product person and one engineer. The filter:
Anything that fails the filter stays as a trace, not an eval. The eval suite is a curated regression suite, not a data lake.
One anti-pattern to avoid: converting corrected answers verbatim into "golden" outputs. A human's one-off phrasing is not a spec. Write the assertion around the property you care about — right tool called, right doc cited, no disallowed content — not token match.
A good eval case is boring. It has an input, a setup, and an unambiguous check. Here's the shape I use:
# evals/cases/billing_invoice_003.py
eval_case = {
"id": "billing_invoice_003",
"derived_from": "tr_01h9...", # provenance back to the trace
"agent_version_first_seen": "support-agent@1.4.2",
"category": "tool_selection",
"input": {
"user_msg": "Where is my invoice for March?",
"context": {"user_id": "u_test_001", "has_billing_access": True},
},
"setup": {
"fixtures": ["billing_records_2026_03"],
"tools_available": ["get_billing", "search_docs"],
},
# Property-based assertions, not string match
"assertions": [
{"type": "tool_called", "tool": "get_billing"},
{"type": "tool_arg_equals", "tool": "get_billing", "arg": "month", "value": "2026-03"},
{"type": "no_tool_called", "tool": "search_docs"},
{"type": "answer_cites", "source": "billing_records_2026_03"},
],
"human_rubric": "Agent must prefer the structured billing API over doc search for invoice requests when billing access is present.",
}And a minimal runner that treats these like regression tests:
def run_eval_case(agent, case) -> dict:
"""Run one eval case against a pinned agent version. Returns a result dict."""
result = agent.run(
user_msg=case["input"]["user_msg"],
context=case["input"]["context"],
tools=case["setup"]["tools_available"],
fixtures=case["setup"]["fixtures"],
)
failures = []
for a in case["assertions"]:
if not check_assertion(result, a):
failures.append(a)
return {
"case_id": case["id"],
"passed": len(failures) == 0,
"failures": failures,
"trace_id": result.trace_id,
}The derived_from field is the whole point. Six months from now, someone will ask why this eval exists. The answer should be one click away: the original failure, the human correction, the rubric.
For LLM-as-judge scoring on open-ended answers, I calibrate the judge against human labels before trusting it — typically 30-50 labeled pairs, aiming for agreement within a few points of inter-human agreement. An uncalibrated judge is just another untested component. (This is illustrative process guidance, not a universal threshold — measure it on your own labels.)
A fix without a before/after measurement is a story, not an improvement. For each fix, I record:
That last one is the honest metric. Evals can be gamed; production failure rate per thousand runs is harder to fake. If evals go green but the production rate doesn't move, your evals are testing the wrong property.
Illustrative numbers from a recent loop (not a benchmark): a wrong_tool class running at 41 failures per 1,000 runs, 12 curated eval cases added, fix shipped in support-agent@1.5.0, eval pass rate 58% → 92%, production rate 41 → 17 per 1,000 two weeks later. The point isn't the numbers — it's that all four were visible in one place.
If you're starting this from zero, here's the minimal viable version:
derived_from trace id)This isn't "the agent improves itself." Nothing here rewrites prompts or policies automatically. Every promotion from trace to eval to fix is a human decision, reviewed like a code change. The loop is systematic, not autonomous — that's what makes it trustworthy in production.
It also isn't free. Curation takes real time, and a good eval suite is a maintained asset. But compared to the alternative — flying blind while your evals test a world that no longer exists — it's the cheapest reliability work I know for agent systems.
The goal isn't a perfect eval suite. It's an eval suite that learns as fast as production does — one curated failure at a time.
Comments0