Flaky Tests: Why Your Automation Flickers and How to Fix It

A familiar picture: a test fails, you rerun it — green. The code didn’t change. Two days later it fails again, on a different step. That’s a flaky test: a non-deterministic result on unchanged code.

The problem isn’t that one test occasionally lies. The problem is that flaky tests destroy trust in the whole suite. When a red build gets “fixed” by a rerun, the team stops reading reports and starts hitting “rerun” on autopilot — and that’s exactly when a real bug slips into production disguised as another flake. Google’s Testing Blog reported that around 16% of their tests show some degree of flakiness; this isn’t “bad tests”, it’s a property of large systems you have to manage.

Let’s break down where instability comes from and what to do about it.

1. Timing and implicit waits

The #1 cause for UI tests. The test clicks a button before it renders; checks text before the API response arrives. Passes on a fast machine, fails on a loaded CI agent.

The main antipattern is sleep(3). It’s either too short (flaky) or too long (slow suite). There’s no magic correct number, because response time isn’t constant.

Wait for a condition, not for time. Explicit wait until a state: “element is clickable”, “spinner is gone”, “list has ≥1 row”. Playwright and Cypress do this via auto-waiting out of the box — don’t defeat them with manual pauses.
Remove sleep from tests. If you really need it, that’s a signal you’re missing an explicit readiness marker (a data attribute, an event, a DOM change).
Wait for the right marker. Not “2 seconds passed”, but “the Submit button became enabled” or “the network went idle”.

2. Shared state between tests

Tests share one database, one account, one cache, a file or a queue. Test A created a record, test B didn’t expect to see it — and fails intermittently depending on who ran first and what’s left over from the previous run.

Isolation. Each test gets its own data: a unique user, its own tenant, a key prefix. Don’t rely on an “empty” database.
Clean state. Setup creates what’s needed; teardown cleans up after itself (or wrap it in a transaction with rollback). “I’ll leave the data, might be useful” is a source of flakiness.
Careful with parallelism. Under parallel runs, a shared resource becomes a race. Either isolate the resource per worker, or explicitly serialize the tests that need it.

3. Order dependency

An adjacent problem: the suite is green only in a specific order. Test 5 works because test 3 logged in earlier and left a session. Change the order (or enable shuffle) and it falls apart.

Every test is self-sufficient. It brings the system to the needed state itself and doesn’t rely on neighbors’ side effects.
Run in random order. pytest-randomly, --order-randomly and friends deliberately break hidden dependencies — better to catch it yourself than at release.
No “global” per-module logins if tests can run separately or in parallel.

4. External dependencies and the network

The test hits a real third-party API, mail, a payment provider. The external service lagged, returned a 503, hit a rate limit — and your “logic test” went red through no fault of its own.

Mock what you’re not testing. Testing your own business logic — stub the external API (WireMock, MSW, the framework’s built-in mocks). Verify the real integration with a separate, small set of contract/E2E tests, and treat their flakiness differently.
Retry at the infrastructure level, not inside assertions. An unstable network is cured by a transport retry policy, not by a try/except around a check.
Don’t depend on “live” data. A test that checks “10 products on the home page” breaks when content changes. Fix your test data.

5. Animations and async rendering

The element is in the DOM but still sliding through an animation; the SPA paints the list in chunks; focus/scroll hasn’t caught up. The click lands “next to” or on an old position.

Disable animations in the test environment. prefers-reduced-motion, a build flag, a global CSS * { transition: none } — the standard trick for stable visual and UI tests.
Wait for stabilization, not appearance. Element visible ≠ element ready. Wait until position/size stop changing, the spinner is gone, the request finished.
Visual tests with a threshold and masking. Mask dynamic zones (time, avatars, animations); compare with a tolerance, not pixel-by-pixel.

6. Non-deterministic data, time and concurrency

The test depends on now(), a random value, the current locale or the agent’s timezone. At midnight UTC or on a machine in a different zone — it fails. Plus genuine race conditions in async code.

Freeze time. libfaketime, freezegun, the framework’s fake clock. Don’t assert “date = today” if “today” depends on the runner’s timezone.
Fix the seed of random data generators so a failure can be reproduced.
Unique data instead of hardcode. user_${uuid} instead of [email protected], which is already taken from a previous run.
Races are a product bug, not a test bug. If a test reliably catches a rare race — don’t silence it with a retry, fix the cause.

How to catch flakes without drowning

You can’t fix what you don’t measure. Separate from the fixes themselves, you need a detection system:

Rerun detector. Run a test N times in a row (or on a CI schedule) — a stably green one passes, a flickering one lights up. Many frameworks and Allure can flag flaky by history.
Quarantine, not deletion. Move a caught flake into a separate group that doesn’t block the release but doesn’t vanish from sight either. Deleting means losing coverage; keeping it in the main suite means training the team to ignore red.
Don’t hide retries. An automatic retries: 2 in the runner masks the problem: the build is green while the test still flickers. A retry is acceptable as a temporary crutch with mandatory tracking of the retry count — otherwise flakes accumulate invisibly.
Instability dashboard. A “flaky rate” metric per suite + the top flickering tests. It’s a fix queue, like a bug tracker.

Checklist

No sleep(N) in tests — only condition-based waits.
Each test creates its own state and cleans up; no shared accounts/data.
The suite passes in random order (shuffle/randomly).
External services are mocked; “live” integrations are a separate small set.
Animations are off in the test environment; wait for stabilization, not appearance.
Time is frozen, seed is fixed, test data is unique.
There’s a flaky detector (rerun/history) and quarantine for caught ones.
Runner retries only with retry-count tracking, not silently.
There’s a flaky-rate metric and a fix queue.

A flaky test isn’t “bad luck”, it’s a concrete cause: a race, shared state, time, the network. Find it once and fix it — otherwise you’ll keep “curing” it with reruns right up until the day a real bug is hiding behind the next flicker.