AI Agent Evaluation Design

Reading map

Contents

Preface: Who This Book Is For and What It Will Teach

1. Part I: Understand the System and the Evaluation Problem

   1. The System Around the Agent

      Map the model, agent harness, tools, authority, and outside state that shape an agent's behavior. Learn what a trace can show and where evaluation coverage begins and ends.

   2. What Evaluation Design Involves

      Start with the agent's intended job, then choose situations that test its requirements and the evidence needed to judge them. See how those design choices differ from the evaluation harness that runs the cases.

2. Part II: Design the Evaluation Before Running It

   3. Turning Requirements into Evaluation Criteria

      Turn intended behavior into criteria before looking at results. Define permissions, acceptable alternatives, prohibited actions, and the acceptance bar with the person responsible for it.

   4. Designing Cases and Test Conditions

      Build AI agent evaluation cases with a starting state, permissions, tool responses, expected outcomes, and evidence to inspect. Include representative, boundary, and targeted failure conditions.

   5. Designing and Calibrating Judges

      Choose direct checks, rubric-based model judgments, or human review according to the criterion and available evidence. Calibrate on reviewed examples and keep insufficient evidence distinct from failure.

3. Part III: Run, Inspect, and Maintain the Evaluation

   6. How an Evaluation Harness Runs

      Run versioned cases from controlled starting states, capture actions and outcomes, and route evidence to the right checks. Keep each run's status, judgments, and versions inspectable.

   7. Interpreting Results and Making Decisions

      Read results by requirement and failure type. Separate agent failures from evaluation defects, judge disagreement, and missing evidence before making a decision from the tested cases.

   8. Keeping Evaluations Useful as the System Changes

      Preserve a baseline when models, prompts, tools, or requirements change. Compare versions under matched conditions and turn production observations into reviewed regression cases.

Closing

About the Author

Preface

Who This Book Is For and What It Will Teach

What would you need to show before relying on your agent for a particular task?

You may already have tests, recorded runs, and examples where it does what you want. The next question is what those checks establish. Which requirements did they examine? Under what conditions? Could they detect the failures that matter? Who decided that the results were acceptable?

This book explains how requirements become evaluation cases, how checks use evidence, and what a result can tell us. By the end, you should be able to see why a judgment follows from a record and where its limits are.

Who this book is for

If you are a solo founder building with AI, you may be responsible for both the agent and the decision to use it. You need a manageable way to start evaluating it, along with enough understanding to recognize when the problem needs more expertise or evidence than you can provide alone.

If you are an AI lead, you may have engineers building the system and evaluations already running. You need to assess whether those evaluations examine the right behavior, whether their judgments are reliable, and who is responsible for resolving the gaps. The examples give you concrete decisions to inspect and challenge.

Testers and evaluators should also find the material useful. The book assumes basic programming, familiarity with software testing, and an understanding of how agents use tools. It assumes no previous evaluation-design experience. Necessary terms will be explained as they appear.

Your existing testing knowledge gives you a useful starting point. We will build on it by making requirements, case conditions, evidence, and judgment explicit for an agent system.

What you will learn

We will follow the main decisions in an evaluation:

1. Understand what behavior is required and who sets that expectation.
2. See how cases put the requirement to a useful test.
3. Identify what a run would need to show.
4. Learn how checks use the record to reach a judgment.
5. Interpret what that judgment supports and what remains unknown.

Each step contains decisions worth understanding. A case needs more than an input message if permission, prior state, or a tool response affects the correct action. A judgment needs a clear basis. The method making it must distinguish acceptable behavior from a failure and recognize when the evidence is insufficient to decide.

I use a small incident-response agent as a recurring example. It investigates a checkout failure, can request a rollback under defined conditions, checks whether ordering works, and hands off when it cannot resolve the incident. Its limited setting keeps some evaluation questions concrete. When another example explains a concept more simply, we will use it.

Following the book

The chapters move from understanding the system to designing checks, seeing how an evaluation harness uses them, and interpreting results. Each step gives the next one a purpose.

Use the short checkpoints to test your understanding of the examples. Where several answers are valid, the reasoning matters.

Figures and short example records will make the mechanisms easier to see.

Before we design checks, we need to know which parts of the agent system can affect the behavior we see. We begin by mapping those parts, then introduce the choices involved in evaluation design.

Chapter 01

The System Around the Agent

When you say "I'm evaluating my agent," what exactly are you putting to the test?

The model produces a response, but your application decides what information it receives, which tools it can use, and how a requested action reaches the outside world. A change in any of those parts can change the behavior you observe. To evaluate that behavior, you need to know where your system begins and ends.

This chapter shows how the agent harness runs a task, what a trace can tell us about the run, and how the evaluation harness tests selected behavior.

The shop, the incident, and the agent's task

Our sample shop sells coffee beans and brewing equipment. A customer adds coffee to their basket, enters delivery details, and clicks Place order. Normally, checkout creates an order and shows a confirmation.

At 10:00, a deployment changes address storage from one delivery_address field to separate street, city, and postal-code fields. The migration removes the old field, but a leftover order-creation path still writes to it. Checkout now fails before creating an order or requesting payment. Customers can still browse and add items to their baskets.