AI Agent Evaluation
AI Evals
Everyone agrees you should evaluate an AI system. Far fewer people agree on what an evaluation is supposed to prove, and that is where most of them go wrong.
What are AI evals?
An AI eval is a test of whether an AI system did what it was required to do. Not whether the output looked reasonable, and not whether a reviewer liked it. Whether it met a requirement someone wrote down in advance.
That last clause is the whole of it. An eval is made of four things: the criteria that say what acceptable behavior is, the cases the system is run against, the expected behavior that says what a correct result looks like, and the evidence left behind so someone who did not build the system can check the claim themselves.
Take any one of those away and you still have a number. You no longer have an evaluation.
What an Eval Is Not
- Not a benchmark. A benchmark compares systems on a shared task. An eval asks whether one system meets your requirement, and your requirement is not on any leaderboard.
- Not a score. A score is the output of an eval, not the eval. Ninety-four percent of what, measured against which expected answer, is the question that decides whether the number means anything.
- Not a vibe check. Reading twenty outputs and deciding they look fine is a reasonable thing to do on a Tuesday. It is not repeatable, so it cannot tell you whether Thursday made things worse.
- Not a unit test. A unit test asserts one exact answer. An AI system has a range of acceptable answers, so the eval has to encode the range, which is a design problem before it is an engineering one.
Where an Eval Actually Starts
Most eval work starts at the output and reasons backwards. That is the wrong end. You cannot judge a result until you have said what a correct one would be, so the work runs in this order:
Requirement
What the system is actually supposed to do, written down.
Evaluation criteria
What counts as acceptable behavior, stated before you look.
Test cases
The situations it has to handle.
Edge cases
The situations where the requirement is most likely to break.
Expected behavior
What a correct result looks like, so a result can be judged.
Evaluation
How the system actually performed against all of it.
Evidence
What you can show someone who did not build it.
Skip the first step and everything below it inherits the gap. The criteria measure the wrong thing, the cases test the wrong behavior, and the score comes back green on a requirement nobody ever wrote down. That failure is the reason this practice exists.
Running Them More Than Once
An eval you run by hand once is a report. An eval you can re-run after a prompt change, a model upgrade or a new tool is infrastructure, and building that is a discipline of its own: harness engineering. That is the rung above this one.
Read Next
The same subject, taken one decision at a time.
Is Your AI Agent Eval Set Actually Testing Anything?
Most teams treat AI agent evals as an afterthought. Your eval set is the product: how to build failure cases that survive every model swap.
Who Should Own AI Evals on Your Team?
AI evals get filed as an engineering chore. Four of the decisions inside them are product decisions, and handing those to engineering is how quality drifts.
How Do You Build an Evaluation Harness for AI Agents?
An evaluation harness for AI agents is four decisions, not a framework. Where the cases come from, what a rubric must survive, who judges, where the bar goes.
How Do You Measure AI Agent Reliability?
An agent won't give the same answer twice, so one green eval run is a coin flip. How to measure AI agent reliability as a distribution, not a pass.
Is Your AI Agent Production-Ready? Define the Bar First
"Is it ready?" is a circular meeting until you define the bar. The four lines that make a production-ready AI agent a number, not a vibe.
We design evaluations and build the harnesses that run them. See how that works, or tell us what you need to prove.