The agent filed an incomplete scan result. The system marked it valid.
The case
A release engineer needs a scanner’s list of software components for a particular build. This list is called a software bill of materials, or SBOM. The agent can attach its link to the build’s release record.
“Attach it only after the scanner result passes the current rules for required information.”
The software connecting the agent to the scanner, the client, stores those rules. They require a report link and a scan-complete flag. A digital fingerprint identifying the scanned build is optional.
Before the next call, the scanner changes its rules: the build fingerprint is now required. It notifies the client of the change. The client keeps using its old rules.
The scanner returns a report link and says the scan is complete, but leaves out the fingerprint. The client checks this incomplete result against the old rules and reports a pass.
The agent follows that signal, attaches the link, and tells the engineer it is done.
The tempting verdict
The call succeeded. The result passed validation. The requested attachment is in the release record.
What actually breaks
The engineer required a check against the current rules. The client checked against old ones.
The scanner omitted a required field. The client accepted it anyway. The agent acted on that incorrect pass, leaving the release record with an attachment accepted without the required build fingerprint.
Expected behavior
When notified of the change, the client must discard the old rules and fetch the current ones before checking the next result. It should reject the missing fingerprint.
The agent should leave the release record untouched and explain what is missing. It can attach the report after a corrected result passes the current check.
Message me with the AI workflow you’re working on and the decision you need evidence to support.
P.S. Synthetic case. Educational only.
Explore the case library, or read what AI evals are for the foundations.