Harness Engineering Series #1
How Do You Know Your AI Agent Actually Works?
Harness engineering, explained: why your tests pass while production breaks, and how to build the evaluation set that catches it. Free guide.
Harness Engineering Series
Harness Engineering
The part of the job that starts after the tests pass, and the reason an agent that passed every one of them still broke on Monday.
What is harness engineering?
Harness engineering is the discipline of building the machinery around an AI agent rather than the agent itself. The prompt is usually about forty lines. Almost everything that decides whether the thing can be trusted sits outside it.
The name is borrowed from how engineers have always handled something powerful they cannot fully predict. Nobody takes an engine's word for its own output. They build a test harness around it, with instruments on what comes out, load on what goes in, and alarms on every limit that matters. The engine supplies the power. The harness is what lets you say you know.
An agent is an engine in exactly that sense. It is non-deterministic, it is persuasive when it is wrong, and it will answer the same question five different ways. A test suite written by the same person who wrote the prompt cannot see any of that, which is why 340 passing tests on a Friday tell you nothing about the 43 tickets waiting on Monday.
Teams believe the job is prompting, and the market sells them the engine. But the engine was never the hard part. You cannot trust what you cannot measure, and nothing about an agent is measurable until you build the thing that measures it.
A harness is what you build once you know what you are testing for. If that part is still open, start with what AI evals are and come back.
What a harness actually contains
- An evaluation set that production keeps growing, built from things real users actually sent.
- Rubrics precise enough that two judges reading the same answer land on the same score.
- Judges: code where the answer is checkable, humans where it is a judgement, and an AI judge only once it agrees with the humans.
- A bar chosen from what a failure actually costs, not from a number that looked respectable.
- A step budget and a cost meter, because an agent that loops quietly is still failing.
- An alarm on the behavior distribution, so you find out when it moves rather than when a customer does.
The guides
Long form, free to read in full, and written for the part of the job that happens after the tests pass.
Coming next
- Building Your First Evaluation Dataset
Shorter pieces on the same problem
Each one takes a single piece of the harness and goes at it on its own.
- What Is Agentic Engineering, and Where Does It Break?
Agentic engineering is building the machinery around the model, not the model. What the term means, the four places it breaks, and the half nobody budgets for.
- Who Should Own AI Evals on Your Team?
AI evals get filed as an engineering chore. Four of the decisions inside them are product decisions, and handing those to engineering is how quality drifts.
- Is Your AI Agent Eval Set Actually Testing Anything?
Most teams treat AI agent evals as an afterthought. Your eval set is the product: how to build failure cases that survive every model swap.
- Is Your AI Agent Production-Ready? Define the Bar First
"Is it ready?" is a circular meeting until you define the bar. The four lines that make a production-ready AI agent a number, not a vibe.
- How Do You Measure AI Agent Reliability?
An agent won't give the same answer twice, so one green eval run is a coin flip. How to measure AI agent reliability as a distribution, not a pass.
- How Do You Contain an AI Agent Failure You Can't Prevent?
You can't stop an agent from ever being wrong. AI agent guardrails, human-in-the-loop, and blast radius: how to contain the failure you can't prevent.
- What Does Your AI Agent Really Cost to Run?
An accurate agent still dies in review if it is too slow or too expensive. How to budget AI agent cost and latency per successful task, not per call.