Autotests that aren't in CI are a hobby: wiring tests into the pipeline without drowning
I’ve seen more than one team with a decent set of autotests that run… sometimes. Locally, on the automation engineer’s machine, “before a big release.” Let me put it harshly, the way I once phrased it for myself: an autotest that doesn’t run in CI on every change isn’t automation — it’s a hobby. An autotest’s value is an early, reliable “it broke” signal, and it can only signal from the pipeline. Everything else is code that ages.
At the same time, “just turn the tests on in CI” is a great way to drown: hour-long runs, flaky tests killing every second build, developers hitting retry without looking, and a month later the red pipeline is background noise nobody sees. Here’s what a working integration is made of.
Layers: not everything runs on every push
The first mistake is running everything on every commit. A full e2e regression per PR means hour-long feedback, while the whole point of continuous integration is the opposite: learning quickly that your change didn’t break anything. A slow gate stops being waited for — and starts being worked around.
I split runs into three layers. On every PR — the fast stuff: unit tests, component tests, API tests, a smoke pass over critical paths; the budget is minutes, not tens of minutes. If the PR gate takes longer than ~10 minutes — trim its contents, don’t tolerate it. After merge to main — the core regression: what’s too slow for a PR but must be caught before new changes pile on top. Nightly/scheduled — the heavy and slow: full e2e, cross-browser/devices, load tests, visual regression across all pages.
The placement criterion is simple: the earlier the layer, the faster and more stable its tests must be. A slow or flaky-ish test isn’t “thrown out” — it moves one layer later.
Parallelism: tests kill each other, CI gets the blame
The only way to keep a large suite fast is parallelism and sharding: Playwright has --shard out of the box (and is generally well documented for CI), pytest has pytest-xdist. But parallelism mercilessly exposes the sin I wrote about in the test data post: dependent tests and shared data. Two shards edit the same user simultaneously — both fail, differently, only under CI load and never locally. It looks like “CI is glitchy”; it is “the tests are coupled.”
The rule: before turning up parallelism, make sure the tests pass in random order and with unique data per worker. Otherwise parallelism won’t speed you up — it will bury trust.
Retry: insurance that easily becomes a drug
Automatically retrying a failed test is reasonable insurance against a network hiccup. But retry without accounting is a flakiness-masking machine: a test blinks for months while the pipeline stays green. Nobody sees the degradation until it’s total.
My rule: retry at most 1–2 times, and every passed-on-retry gets recorded — in the report, in a metric, in the flaky-candidates list. Passing on the second attempt isn’t green, it’s yellow: the test falls under suspicion, and “yellow” has an owner and a triage deadline. In the “what to automate” post I wrote that a flaky test is worse than a missing one — retry without accounting is precisely the mechanism that lets flakiness live forever.
Quarantine: flaky tests get isolated, not muted
What do you do with a blinking test right now, before it’s fixed? The worst option is commenting it out/muting: the coverage hole becomes invisible and permanent. The best workable one is quarantine, as the Google Testing Blog described in its flaky tests write-up: the test keeps running, its result doesn’t affect build status, and the quarantine itself is a ticket with an owner and a deadline.
The key part — quarantine has exactly two exits: the test is fixed and returns to duty, or it’s judged not worth maintaining and deleted. Quarantine without a deadline is the same graveyard with a nicer name. A useful limit: no more than N tests in quarantine at once (say, 10) — overflow means the suite has a systemic problem, and that problem matters more than new features.
Red main: the rule without which everything else is pointless
Technically you can build a perfect pipeline and still arrive at “red is normal, it’s always like that.” Only a social contract saves you: a red main gets fixed immediately or the change gets reverted. Not “I’ll look after lunch” — immediately, because new changes get merged on top of the red one, and a day later nobody can tell whose fault it is. A rotation works well: a build sheriff of the week who triages failures — not fixing everything personally, but guaranteeing every red has an owner. And the hard rule: no merging into a red main with “well, my tests passed.”
Honesty must go both ways: if a test failed because of infrastructure (the environment died, the docker registry was unreachable) — that’s a signal too, about pipeline reliability, and it needs an owner too. “Infra flakiness” corrodes trust just like the test kind.
A failure without artifacts = debugging all over again
A test fails in CI, and then — “doesn’t reproduce locally,” and the automation engineer spends half a day re-running it by hand. The cure is artifacts: on failure, save the screenshot, video, trace (in Playwright — trace: 'on-first-retry', opened in the trace viewer with every action, DOM snapshots and network), and application logs for the test’s duration. Artifacts attach to the CI run; the report lands right in the PR. A good failure reads without a re-run: what the test did, what it saw, what came back from the API.
Plus history: a single report says “what failed now,” while run history (Allure, ReportPortal, built-in CI dashboards) says “what fails most often” and “when the suite started degrading.” The month’s top-10 flaky tests are an automation engineer’s most honest backlog.
Environment: don’t point CI tests at shared staging
CI tests that hit a shared staging — where people click around and other pipelines run in parallel — are a source of “mysterious” failures: data shifts underfoot, the code version on the stand and in the PR drift apart. The ideal is an ephemeral environment per run: docker compose / Testcontainers bring the app and its dependencies up inside the pipeline, run, tear down. If the project can’t afford that — at least a dedicated environment reserved for autotests, with exactly the version under test deployed.
Suite health metrics
To see degradation early instead of “CI’s gotten really bad somehow,” four numbers on a dashboard suffice: run duration (trend), flake rate (% passed-on-retry), time-to-green (how long a red main lives before it’s fixed), and quarantine size. Any of them growing is an actionable signal, not a feeling.
The “tests in CI” checklist
- Layers: a fast PR gate (minutes) / post-merge / heavy nightly
- PR gate longer than ~10 minutes — revisit its contents, don’t tolerate it
- Tests pass in random order with unique data — before enabling parallelism
- Retry 1–2 max, passed-on-retry recorded as a flaky candidate
- Quarantine: keeps running, doesn’t affect status, ticket with a deadline, size limit
- Red main: fixed immediately or reverted; a sheriff rotation; no merging onto red
- Artifacts on failure: screenshot, video, trace, logs — the failure reads without a re-run
- Run history and top flaky tests — the trend is visible, not just current status
- Ephemeral or dedicated environment, not shared staging
- Dashboard: duration, flake rate, time-to-green, quarantine size
A pipeline with tests is a product with its own users: developers who await its verdict ten times a day. The product has UX (feedback speed), reliability (flakiness), and support (the rotation, the quarantine). Treat it worse than a product for external users — and don’t be surprised people stopped using it.