How Do You Build Your First Eval Set for an Agent?

Someone asks whether the AI feature has got worse, and the room cannot answer. Not because the team is careless. Because good has never been written down anywhere, so everyone in the meeting is checking the agent against a private version of it. An eval set is where you stop doing that.

You do not need a platform to build the first one. You need a spreadsheet and an afternoon.

What goes in your first eval set

Open the logs and pull twenty real conversations from the last month. Not the clean ones. The ones somebody forwarded with a comment attached, the ones that made a colleague uncomfortable, the ones that were technically correct and still felt wrong.

Put each one in a row. Next to it, write the answer you would have been happy to ship.

That is the whole artifact. Twenty inputs, twenty answers you stand behind. It is small on purpose. Twenty rows you have actually thought about are worth more than two thousand you generated and never read.

Then hand it to someone else

Give the same twenty conversations to another person on the team. Product, support, the founder, whoever gets the call when the agent says something strange. Ask them to write their answer next to each one without seeing yours.

Your answers will not match.

Every row where you disagree is a product decision nobody has made yet, and the agent has been making it on your behalf in the meantime. How much warmth is worth how much accuracy. Whether a confident wrong answer is worse than a vague right one. When refusing is safe and when refusing is just annoying. Whether close enough ships.

Those questions arrive dressed as engineering questions, because they surface while somebody is writing a grader. They are not. The person who should answer them owns the product.

Settle the disagreements one row at a time, and write down the reason, not only the verdict. The verdict covers that row. The reason covers every row like it, which is the part you reuse.

Now pick a tool

With twenty rows, agreed answers and stated reasons, you have something a tool can run. Hamel Husain's field guide to evals is the best walkthrough of the mechanics. The choice of runner matters less than people expect, because they all do the same job: apply your definition of good, over and over, without getting tired.

The four decisions that keep surfacing in those rows are written out at length in our free guide, How Do You Know Your AI Agent Actually Works?, if you want the longer version.

Most teams do this in the opposite order. The platform arrives, the numbers arrive, and the room still cannot say what the numbers are for. A score with no agreed definition underneath it is a number that moves. It is not evidence.

This is the part of harness engineering that has nothing to do with infrastructure. The harness is the machinery. The definition is what you feed it, and the definition is where the work actually is.

The honest limit

Twenty rows will not tell you your agent is good. There are far too few of them, they came from one month that may not resemble next month, and they carry the assumptions of the two people who wrote them.

What they will tell you is whether it changed. Freeze the set, run it after every prompt edit and every model upgrade, and you get a before and an after on the same questions. "Did we regress" is answerable with twenty rows. "Is this good" is a longer and more expensive conversation, and nobody should sell you a dashboard that pretends otherwise.

Start with the twenty. The argument you have on the way there is worth more than the file you end up with.

Designing that definition with teams is the work we do at nugalaxy.