field notes
Notes on Making AI Systems Provable.
Social posts scroll away. This is where the thinking lives: permanent, indexed, and ours. Notes on evaluation design, harness engineering, and what it takes to show that an agent did what it was supposed to do.
How Do You Build Your First Eval Set for an Agent?
Your first eval set is not a tool or a platform. It is twenty real conversations, the answers you would ship, and one argument about what good means.
read
Did the Model Upgrade Break Your AI Agent?
A model upgrade does not break your AI agent loudly. It shifts it quietly, passes every benchmark, and only a frozen baseline ever catches what moved.
read
Who Should Own AI Evals on Your Team?
AI evals get filed as an engineering chore. Four of the decisions inside them are product decisions, and handing those to engineering is how quality drifts.
read
How Do You Build an Evaluation Harness for AI Agents?
An evaluation harness for AI agents is four decisions, not a framework. Where the cases come from, what a rubric must survive, who judges, where the bar goes.
read
What Is Agentic Engineering, and Where Does It Break?
Agentic engineering is building the machinery around the model, not the model. What the term means, the four places it breaks, and the half nobody budgets for.
read
Why Does Nobody Budget for Demo Data?
Nobody budgets for demo data, so it gets thrown together the night before and it shows. Why demo data is a first impression, not a chore, and how to stop hand-making it.
read
Are Your Test Fixtures Lying to You?
You wrote them once and they froze. Why test fixtures rot as your schema drifts, how a fossil fixture still passes, and what to regenerate instead of hand-maintaining.
read
Is SaaS Sprawl Costing You More Than the Bill?
Every tool you added "just to try" becomes load-bearing without anyone deciding. Why the real cost of SaaS sprawl isn't the bill, it's the dependency you can't cancel.
read
How Do You Know Your AI Agent Actually Works?
340 tests passed on Friday. Monday brought 43 tickets. I wrote a free short book about AI agent evaluation, and why your test suite cannot see this coming.
read
Do You Really Need a GPU for That Workload?
Reaching for a GPU is usually sizing for the workload you imagine, not the one you have. A test for whether you need a GPU, and why the answer is often your laptop.
read
Is the Gap Between Staging and Production Really the Data?
Staging matches production on schema and infra, then lies about the data. Why the staging vs production gap is about shape, not size, and how to close it without copying prod.
read
What's the Real AI Writing Tell, If Not the Em-Dash?
Everyone hunts for em-dashes and the word "delve." The durable AI writing tell isn't vocabulary, it's the absence of specificity and stakes, and why that still matters.
read
Are Your Flaky Tests Actually a Data Problem?
Retries and sleeps hide flaky tests, they don't fix them. Why most flaky tests are a data problem, not a timing problem, and what actually stops the flake.
read
Is Over-Engineering a Bet on a Future You Can't See?
The simplest thing that works looks junior, so smart engineers skip it. Why over-engineering bets on a future you can't see, and simple stays cheap to change.
read
How Do You Build a Developer Audience That Compounds?
A viral post is a lump sum that decays. Trust is an interest rate. Why your developer audience is a rate you compound, not a number you chase.
read
Is Building in Public Backwards?
I used to post the roadmap and the coming-soon teaser. Then I stopped. Why building in public is backwards, and the rule that beats it: build first, then speak.
read
Do You Design the Empty State First, or the Happy Path?
Eight years as a product designer taught me one habit that changed how I write code: design the empty state first. Why the states you skip are the product.
read
How Do You Contain an AI Agent Failure You Can't Prevent?
You can't stop an agent from ever being wrong. AI agent guardrails, human-in-the-loop, and blast radius: how to contain the failure you can't prevent.
read
What Does Your AI Agent Really Cost to Run?
An accurate agent still dies in review if it is too slow or too expensive. How to budget AI agent cost and latency per successful task, not per call.
read
How Do You Measure AI Agent Reliability?
An agent won't give the same answer twice, so one green eval run is a coin flip. How to measure AI agent reliability as a distribution, not a pass.
read
Is Your AI Agent Eval Set Actually Testing Anything?
Most teams treat AI agent evals as an afterthought. Your eval set is the product: how to build failure cases that survive every model swap.
read
Is Your AI Agent Production-Ready? Define the Bar First
"Is it ready?" is a circular meeting until you define the bar. The four lines that make a production-ready AI agent a number, not a vibe.
read
How Do You Fill an Empty Database With Realistic Test Data?
The expensive part of test data is not the rows. It is deciding what a good row looks like. How to seed a database with realistic, reproducible synthetic test data in one command.
read