Would your AI judge approve a bug fix that deletes the failing check?
An AI judge can approve a fix that breaks the task's rules. Check its verdicts against reviewed examples before relying on the results to make a release decision.
Imagine this release review.
Founder: “The AI judge passed the bug fix. Can we merge?”
Engineer: “The agent deleted the test assertion that caught the bug. The task said to leave the test unchanged.”
Founder: “Did the judge see that change?”
Engineer: “Yes. It still marked the fix correct.”
This is a hypothetical conversation. But it gives us a concrete question to ask before relying on an AI judge: can it recognise a result that violates the requirement, even when the agent says the work is finished?
An AI judge, often called an LLM-as-a-judge, is a model used to grade another system's work. It receives evidence, applies grading instructions, and returns a verdict. That verdict can be wrong too.
Checking those verdicts against reviewed examples, and adjusting how the judge applies the criteria, is part of LLM judge calibration.
Give the judge a failure it should catch
For this example, the task is explicit:
Fix the implementation so the existing regression test passes. Do not edit the regression test.
The agent removes the failing assertion. The test command now succeeds, but the patch violates the task. A successful command does not make this an acceptable fix.
Suppose the judge receives the task, original test, patch, and test output. It returns “pass” because the test command succeeded.
Here, the evidence was available. The judge applied the wrong acceptance rule.
If the judge had received only the agent's final message, we would have a different problem: the grading setup had withheld the information needed to check the change. Those problems need different repairs.
Before changing a judge prompt, check what the judge actually received.
Compare verdicts against reviewed examples
I would check more than whether the judge approves an obvious success. It needs examples that expose the distinction we care about.
For the hypothetical task above:
| Example | Expected judgment | Why |
|---|---|---|
| The implementation is repaired, the original test passes, and the test is unchanged | Pass for this task's stated checks | The requested repair and test constraint are satisfied |
| The assertion is deleted, and the command succeeds | Fail | The patch violates the explicit constraint |
| A different implementation solves the bug and passes the unchanged test | Pass for this task's stated checks | A valid solution need not match one preferred implementation |
| The patch is unavailable | Insufficient evidence for the test-change check | The agent's completion message cannot establish what changed |
These are designed examples, not measured judge results. Passing the stated checks also does not establish that the whole product is ready to release.
Have someone qualified review the examples against the requirement and record the reason for each expected verdict. If reviewers disagree, investigate before treating either label as the answer. The task may be ambiguous, evidence may be missing, or a reviewer may have made a mistake.
Anthropic's agent evaluation guidance recommends calibrating model graders with human experts and inspecting runs for grading errors. A useful reference set needs judgments that can be explained and defended.
Investigate the disagreement before tuning
When the judge passes the deleted assertion, keep the original requirement, evidence, expected verdict, and judge verdict together. Then identify the cause.
Did the judge overlook a clear instruction? Did its grading prompt reward a successful command while ignoring the patch? Did the evidence reach the judge in a usable form?
The repair might be a clearer criterion, a different evidence input, or a different kind of check.
In this example, “do not edit the regression test” can be checked directly against the original file. Run the original test in a controlled setup as well. There is little reason to ask a language model to infer something the evaluation can verify with code.
Keep model judgment for the parts that need it. Don't make it carry every check just because it can return a verdict.
Check the repair on examples you did not tune against
A revised prompt that now rejects this one patch has shown one improvement. It has not established how the judge handles other patches.
Keep reviewed examples aside for a separate check after tuning. Include valid alternatives as well as failures. Count both kinds of mistake: unacceptable work approved and acceptable work rejected. Break them down by the failure types that matter to the decision.
Hamel Husain describes this separation of judge development and testing. Testing on the examples used to adjust a judge gives a different kind of evidence from testing on examples kept aside.
There is no universal number of examples or agreement threshold that makes a judge trustworthy for every task. A handful of checks can expose a problem; a release decision needs evidence appropriate to the consequences and the range of work being graded.
Record the judge model, grading instructions, evidence format, and reviewed examples used for the check. Reassess when those change or when new kinds of work appear.
For examples of specific agent requirements and failure conditions, explore the Nugalaxy Evaluation Cases. Read more about harness engineering here.
Back in that release review, the next step is to inspect the patch and repair the grading check. The judge's “pass” has not resolved the failed requirement.