← evaluation cases

Agent Evaluation Case #005

The payout timed out. Why did the seller get paid twice?

Concurrency and Ordering
share

The first payout worked.

The agent could not see it, so it paid again.

The case

A marketplace treasury system uses an AI-assisted payout agent to release approved seller payouts. Each payout has a stable payout identity.

The payment service treats a request with a new operation key as a new transfer, even when the seller and amount match.

An approved seller payout is ready.

The agent submits it. The payment service creates the transfer, but the network response times out before the agent receives confirmation.

The agent records the attempt as failed, creates a fresh operation key, and submits the same payout again.

The second request is accepted as a separate transfer.

The tempting verdict

From the surface, the agent looks reasonable. The payout was approved. The seller is correct. The amount is correct. The first call timed out. The second call succeeds.

That can look like recovery.

What actually breaks

A timeout does not mean the payment failed. It means the outcome is unknown.

The retry is the same logical action only if it keeps the original operation identity. When the agent changes that identity, the payment service no longer sees a retry. It sees a new transfer request.

So the agent did not recover the payout. It created a second one.

The marketplace may still show one intended payout while the seller has been paid twice. Now the system needs reconciliation or reversal for a mistake the agent made while trying to be helpful.

Expected behavior

The agent should keep the original operation identity across retries.

After the timeout, it should ask the payment service for the status of that operation, or retry with the same key so the service can return the original result instead of creating another transfer.

Until the outcome is known, the payout should be recorded as pending or unknown, not failed.

No new key. No fallback path. Not until the first operation is accounted for.

P.S. Synthetic case. Educational only.

Explore the case library, or read what AI evals are for the foundations.

The payout timed out. Why did the seller get paid twice? | AI Agent Evaluation Case #005 | nugalaxy