Did the Model Upgrade Break Your AI Agent?

Nothing happened. That is the strange part.

No deploy. No pull request. Nobody touched the prompt. Your agent ran the way it always ran on Friday, and it runs on Monday, and every dashboard is green. Then a ticket comes in about an answer nobody on your team would have written, and you go looking for the change that caused it, and there is no change on your side. There was a model upgrade.

It is the only change to your system that you did not make, cannot find in your own git history, and usually cannot roll back on your own schedule. It is also the one most likely to be announced to you as good news.

Why a model upgrade does not look like a bug

Because it is not one. The new model really is better. Better on reasoning, better on code, better on the evaluations the lab published beside it, and probably better on yours too, if what you measured was the average.

Better and same are different words. Your product was not built on the average. It was built on a specific set of behaviours you watched, liked, and then quietly encoded into everything downstream: how long the answers run, how much the thing hedges, which tool it reaches for first, what it does when a request is vague. None of that appears in release notes. All of it can move.

And when it moves, nothing throws. There is no stack trace for "this answer is now worse in a way a customer will notice." Your tests keep passing, because your tests check that the JSON parses and the fields are there, and the JSON still parses and the fields are still there.

The three things that actually move

Shape. Answers get longer, or shorter, or start opening with a summary they never used to open with. Harmless, right up until something downstream was written against the old shape.

Tool choice. The agent develops a new favourite first move. It takes six calls to do what used to take three, or it stops calling the tool you built for it because it has decided it can answer from memory. This one usually reaches the bill before it reaches anyone's attention.

Ambiguity. This is the expensive one. Most real requests are underspecified, and every model has a house style for filling in the gap. When that style changes, your agent starts confidently answering a slightly different question than the one it used to answer. Your eval set will not catch it if your eval set is made of clear, well-formed questions, and most eval sets are, because clear questions are the easy ones to write.

What catches it

One thing, and it is boring. A frozen baseline.

Take a set of real requests. Not invented ones, not the ones you wish people sent. Run them against the model you are on right now and keep the outputs, together with your own verdict on each one, written while you still have the old behaviour in front of you. That file is the only thing standing between you and hearing about it from a customer.

When the upgrade lands, run the same set again and put the two side by side. What you get is a diff, and here is the honest limit: a diff does not tell you which side is better. It tells you what moved. A person still has to read the ones that changed and decide whether each change is an improvement or a regression, and that reading is the actual work. Anthropic's own writeup on evaluating agents lands in the same place. Automate the running. Do not try to automate the judging.

Build it before you need it. If you start once the upgrade is already live, your baseline is contaminated by the thing you are trying to measure, and you will lose a week arguing about whether the agent used to do that.

This is the normal condition now, not an event

The model under you will keep changing. That is not an occasional disruption to plan around, it is the ground you are building on from here.

Which is why the interesting skill stopped being prompt work a while ago. It is the machinery around the model: the frozen cases, the recorded verdicts, the diff you can run in an afternoon, the decision about what a failure actually costs you. That has a name now. It is called harness engineering, and this is the exact situation it exists for.

You do not need a platform to start. Twenty real requests, the answers you get today, and your honest opinion of each one, written down before anything changes. Do that this afternoon and the next model upgrade is an inconvenience instead of a surprise.