Flaky tests are a data problem, not a timing problem

The test passed yesterday. It passes on your machine. It failed in CI, so you hit re-run, and it passed. Everyone knows what that means: flaky test. You add a retry, maybe a sleep, and move on. The build goes green and the smell goes with it.

That reflex is the problem. We file flaky tests under timing, blame the async, the race, the slow container, and reach for retries and sleeps. Sometimes that is the cause. Far more often, a flaky test is not confused about timing. It is telling the truth about your data.

What flaky tests are actually telling you

Here is the tell. Run the test by itself and it passes. Run the whole suite and it fails. That is not a timing bug, that is order dependence, and order dependence has one cause: state that outlived the test that created it. The test is not flaky. It is sharing data with its neighbors, and the outcome depends on who ran first.

Once you see it that way, the usual suspects line up and none of them are timing:

• A shared fixture that one test mutates and another reads, so the second passes or fails depending on ordering.
• Rows left in the database from an earlier test because the rollback only sometimes happened.
• A test that asserts on "the most recent record" when a sibling test also writes records.
• A seed that depends on today, so it passes for eleven months and fails on the first of the month, or on a leap day, or in a different timezone in CI.
• Unseeded randomness that collides once in a few hundred runs and looks exactly like a ghost.

None of those get fixed by running the test again. Running it again just reshuffles the order until you get lucky, which is not a fix, it is a coin you agree to keep flipping.

Retry is a painkiller

A retry does not make a test reliable. It makes an unreliable test quiet. You have not removed the shared state, you have decided to tolerate it and pay for it in build minutes and slow erosion of trust in the suite, because a suite that is green "after a couple of retries" is a suite nobody believes. The first thing a retry culture teaches everyone is that red does not really mean red, and once that lesson lands, the tests have stopped protecting you.

The actual fix is not cleverer waiting. It is determinism and isolation: every test owns its data, starts from a known state, and depends on nothing another test left behind. Freeze the clock instead of reading it. Seed the randomness. Give each test fresh, reproducible data instead of a shared pile everyone reaches into. Do that and the flake does not get retried away, it stops existing, because the thing that was nondeterministic is now a fact you control.

This is the same discipline behind seeding a database properly in the first place. Reproducible test data, the kind you can regenerate identically on every run, is not a nicety. It is what makes a test mean the same thing twice.

Next time a test flakes, do not reach for the retry. Ask what data it is sharing, and with whom.

A flaky test is rarely lying about timing. It is usually telling the truth about your data.