AI Agent Reliability: Measuring What Won't Repeat

Your agent passed the eval, so you shipped. The next day a user sends almost the same input and it fails. Nothing changed. You just learned that "it passed" was one sample of a distribution, and you shipped on a coin flip that landed heads.

Part 1 defined the bar. Part 2 built the eval set that measures it. Part 3 is the uncomfortable part: an agent will not give the same answer twice, so a single green run tells you almost nothing about AI agent reliability.

One run measures luck, not reliability

A deterministic function passes a test or it does not, forever. An agent is a random variable. Run the same case ten times and you might see nine passes and one confident wrong answer. A single run showing the pass is not evidence the agent is good, it is evidence you got lucky once. Reliability is not a checkmark. It is a distribution, and you have to measure its shape.

Measure the distribution, not one draw

• Run every case many times. Report a pass rate, not pass or fail. If a case passes 8 of 10 runs, that is an 80 percent case, and you should know before a user does. (This is the idea behind pass@k: score across repeated samples, not one.)
• Care about the floor, not the average. A 95 percent average sounds fine until the 5 percent is the run that issues a refund or deletes a record. The average hides the worst case, and the worst case is what hurts you. Bound it explicitly.
• Separate flakiness from failure. Some variance is harmless, like wording. Some is a real wrong answer. If you grade the property instead of the exact string (Part 2), harmless variance stops counting as failure and the failures that matter become visible.

Reduce variance on purpose

Once you can see the distribution, narrow it where it counts. Lower the temperature on calls that must be consistent. Constrain the output format so there are fewer ways to go wrong. Add a guardrail that catches the disqualifying failure before it reaches the user. None of this makes an agent deterministic, and chasing determinism is the wrong goal. The goal is a distribution tight enough, with a floor high enough, that you can trust it.

So the release gate is not "the eval passed." It is "the pass rate over many runs clears the bar, and the worst case is bounded." A green checkmark from a single run is not a decision, just an anecdote.

Part 4 turns to the other constraint that decides whether an agent ships: what all of this costs, in latency and dollars, at real volume.

Run it once and you measured luck. Reliability is the shape of the distribution, not a single pass.