Debugging Flaky Python Tests: Time, State, Concurrency, and Isolation

Treat flakiness as evidence

A test that passes most of the time has an uncontrolled input. Common sources include shared mutable fixtures, wall-clock assumptions, random values, thread scheduling, test order, network dependencies, and cleanup that runs only on success.

Capture the random seed, failing test name, order, runtime version, and relevant timing. Re-run the smallest failing test repeatedly and then run it alone. If it fails only in a suite, suspect shared process state, global patches, environment variables, ports, temporary paths, or leaked background tasks.

Remove time and ordering assumptions

Avoid sleeps as synchronization. Wait for an observable condition with a deadline, or use a fake clock when time itself is the behavior under test. Tests involving concurrency should coordinate with events, barriers, or queues instead of hoping the scheduler produces a particular interleaving.

Freeze time only with a scoped fixture that restores the real clock. Patch a dependency where the code looks it up, not an unrelated imported symbol. Use deterministic random seeds for reproductions, but retain tests that explore multiple seeds if randomness is part of the system's risk.

Isolation and cleanup

Each test should own its database transaction or schema, temporary directory, server port, and background tasks. Register cleanup immediately after creating a resource so assertion failures do not leak it. Do not mutate a module-level list or singleton between tests unless a fixture resets it reliably.

External services should be replaced with contract-faithful fakes for unit tests and exercised in a separate integration suite. A fake that omits retries, ordering, or timeout behavior can be deterministic but misleading.

Fix root cause, not the test runner

Increasing retries may reduce red builds while hiding a race. A quarantined test still represents unverified behavior. After reproducing, identify the uncontrolled variable, make it explicit or isolate it, and add an assertion for the desired invariant. Record flake rate separately from ordinary failures so reliability work has an owner.

Run the test alone; repeat it; randomize suite order; run under a slower or loaded environment; inspect logs and resource ownership; then add synchronization or isolation at the actual boundary. A fixed sleep usually makes the test slower without making it correct.