How Do You Build an Evaluation Harness for AI Agents?

You have an agent that works. Now someone asks how you know, and the honest answer is that you tried it about thirty times and it seemed fine.

That answer is fine for a prototype. It stops being fine the moment the thing is in front of people who did not build it.

An evaluation harness for AI agents is what replaces "it seemed fine". It is not a framework you install. It is four decisions, and you can make all four this week.

Decision one: where the cases come from

Not from your imagination. This is the one that quietly ruins most attempts.

When you invent test cases, you invent them out of the same mental model you used to write the prompt. You produce clear, well-formed, one-request-at-a-time inputs, because that is how you think about the problem. Real users send three questions in one message, leave out the order number, describe the thing without naming it, and switch language halfway through.

So take the last two hundred real interactions and read them. Pull twenty that made you uncomfortable. Not the twenty that failed, the twenty you were not sure about. Those are your first cases, and the set grows every week from the same source forever.

If you have no production traffic yet, the substitute is somebody else's messages. Support tickets, sales emails, forum questions. Anything written by a person who was not thinking about your agent when they wrote it.

Decision two: what a rubric has to survive

Write down what a good answer contains before you look at what the agent produced. This ordering is not a style preference. Read the output first and you will write a rubric that the output happens to pass.

Then run the only test that matters. Give the same output and the same rubric to a second person and ask them to score it. If you get a 4 and a 2, the rubric is not finished. Keep tightening it until two people who have not spoken land in the same place.

Most teams skip this and go straight to automating the scoring. You can automate a rubric two humans disagree about. You will get a number every night, and the number will mean nothing, and it will take you a quarter to notice.

Decision three: who judges

Three kinds, and the skill is knowing which question belongs to which.

Code judges anything checkable. Did it call the refund API. Is the total correct. Is the JSON valid. Did it stay inside the step budget. This is cheap, exact and boring, and it should cover more of your set than you expect.

Humans judge anything that is a judgement. Was that the right tone for a customer who is already angry. Was the escalation appropriate. You will not scale this, and you are not supposed to. It is the ground truth the other two are measured against.

A model judges only after it has agreed with your humans on cases you already scored by hand. Run it against fifty human-scored examples first. If it agrees, promote it. If it does not, you do not have an AI judge, you have a second opinion with no track record. Tools like openai/evals give you the plumbing for this; they do not give you the agreement, and the agreement is the part that matters.

Decision four: where the bar goes

Pick the passing bar from what a failure actually costs you, not from a number that sounds respectable.

Ninety-five percent is not a standard, it is a habit. If the failure mode is a slightly awkward sentence, eighty is generous. If the failure mode is refunding money that should not have been refunded, ninety-five is negligent and you need a human in the path regardless of the score.

Write the cost down next to the bar. Anyone who later argues the bar should move has to argue about the cost instead, which is a much better argument to have.

The honest limit

A harness does not tell you your agent is good. It tells you when your agent changed, and it tells you which of two versions is better on the things you decided to care about. Everything it cannot see is a case you did not put in the set.

That is still an enormous amount more than you had, and it is the difference between shipping and hoping. The wider discipline this belongs to is called harness engineering, and this is its first working piece.

Twenty cases. One rubric two people agree on. A code check where you can, a human where you cannot. One bar with a cost written beside it. That is an afternoon, and it is the afternoon that makes every week after it measurable.