How to know your AI agent is ready for production

A demo shows an agent can succeed once, in front of you, on a case you picked. Production asks it to succeed on cases you didn't.

The demo is not the evaluation

Every agent looks ready in a demo. You pick three tasks, run them live, and they work — because you know the product, you phrase the input the way the model expects, and you're standing there to notice if something's off. None of that is true for the first thousand real users.

"It works in the demo" is not an evaluation strategy. It's a single data point dressed up as evidence. AI agent evaluation means testing against a range of inputs wide enough to include the ones that break things, scoring the outcome against a fixed standard instead of a gut feeling, and doing it before the agent is live, not after a customer complains. Teams that skip this step aren't skipping paperwork. They're skipping the only mechanism that tells them what "working" actually means at their volume.

Build a golden dataset first

An llm evals framework is only as good as what you test it against. Start with a golden dataset: a set of real inputs, each with a known-correct output, agreed by someone who understands the task — not generated by the model you're testing.

Pull these from actual usage where you can. Support tickets that were resolved correctly. Documents that were classified by a human. Edge cases your team already argues about — those are worth more than the easy ones, because they're where models disagree with each other and with themselves. 200 cases is a reasonable starting size for most B2B agent tasks; fewer than that and you're measuring noise, not performance.

Keep the dataset separate from anything the model or its prompts have seen. The point is to simulate a stranger walking in, not to grade your own homework.

Task-level pass/fail, not vibes

Once you have the dataset, score against it at the level of the task, not the token. "The agent responded politely" is not a metric. "The agent extracted the correct invoice number" is. Define pass/fail per task, run the full dataset on every material change to the prompt, the model, or the retrieval layer, and track the pass rate over time like you would test coverage.

This is where most teams stop, because a pass rate feels like the answer. It isn't, on its own — it tells you how the agent does on average, not what happens when it's wrong. A 96% pass rate sounds fine until you ask what the other 4% costs, and whether the agent knows it's in that 4% when it happens.

Confidence-threshold routing: escalate instead of guessing

That's the second layer: confidence threshold automation. The agent doesn't just answer — it estimates how sure it is, and below a set threshold, it doesn't answer at all. It drafts, flags, and hands the case to a person. This is human in the loop ai agent design done properly: not a human reviewing everything, and not a human reviewing nothing, but a human reviewing the cases the model itself can't vouch for.

The threshold is the whole design. Set it too low and the agent answers everything, including the cases it's guessing on. Set it too high and every case goes to a human, which defeats the point of having an agent.

What we got wrong: a threshold that looked fine offline

We shipped a threshold for one of our agent workflows — Atheena, our feedback-to-engineering-ticket triage system — that tested well against the golden dataset. Offline pass rate was 97%, and the confidence cutoff we'd tuned against that dataset held up: cases below threshold went to a human reviewer, cases above it were routed automatically, and the split looked healthy.

In production it didn't hold. The offline dataset was built from historical tickets, which skewed toward common, well-formed complaints. Live traffic included a category we'd under-sampled — feedback that referenced multiple product areas in one message — and the model was consistently overconfident on exactly those cases. It scored above threshold, went straight through, and misrouted at a rate we didn't catch for six weeks, because nothing in the pipeline was watching for confidence that was high and wrong at the same time.

The fix wasn't a smarter model. It was widening the golden dataset to include multi-topic feedback at the rate we now knew it occurred in production, and adding a second check: agreement between two independent scoring passes, not just the model's own stated confidence. Where the two disagreed, the case dropped to review regardless of the confidence score. Offline metrics only protect you against the inputs you thought to measure.

The real trade-off is workload, not accuracy

Tighten the threshold and errors go down — but so does automation, because more cases fall to a human. That's not a modelling problem to solve away. It's a business decision about where the line sits, and it's rarely the same answer for every task inside the same product.