What to automate and what to leave manual — and why 'automate everything' kills the suite
“Let’s automate everything” sounds right, and within six months it becomes a red suite nobody trusts. Tests fail, the CI on-call reflexively hits “retry,” green pleases no one and red scares no one. That’s when automation turns from a helper into a cost center.
Here’s how I split “what to automate” from “what to leave manual” — and why the second list isn’t smaller than the first.
The test we fixed for half a year when we should have deleted it
We had one E2E test that failed roughly once a week. Every time for a new reason: an animation didn’t finish, someone changed the test data, a third-party service on staging blinked. We fixed it, added waits, retries — and it failed again.
At some point I sat down and asked: what does it even check? Turned out — a scenario that hadn’t changed in a year and wasn’t going to break, and which was also covered by unit tests a level down. We deleted the test. And it got better: the suite went green, trust came back, the time spent on “fixing flakies” freed up. That was the first lesson — not every test deserves to live.
An autotest costs not “to write” but “to maintain”
The main estimation mistake is counting an autotest’s cost as “time to write it.” In reality a test costs to write + run + fix — for years. It slows every CI run, it breaks on refactors, it needs updating when the UI changes. If the total maintenance is more expensive than the value it brings, it’s not an asset, it’s a liability. Automation pays off on repetition: what runs a hundred times, yes; what’s checked once, almost never.
What I automate
Good candidates converge on one point: stable, frequent, expensive by hand, and deterministic.
- Regression of critical paths — what must not break (login, payment, the product’s key scenario).
- API contracts and calculations — deterministic input/output, cheap and stable to check.
- Cross-browser/cross-device smoke — stifling by hand, easy for a machine.
- Boring repetitive checks that run on every build.
What I leave manual (and that’s not weakness)
- Early UI that’s redone every sprint — automating a moving target means rewriting tests instead of working.
- One-off checks — faster to verify by hand than to write a test you’ll run once.
- Subjective things: “pretty/convenient/not jarring,” animations, copy tone — that’s a human eye, not an assert.
- A rare case with a monstrous setup — if the prep costs a day for one run, do the math.
- Exploratory testing — an autotest checks what you already thought of; new bugs are found by a human.
The pyramid isn’t dogma, it’s a calculator
The test pyramid is easier to treat not as “the correct shape” but as a decision tool: the higher the level (E2E), the more expensive the run and the more fragile the test; the lower (unit/API), the cheaper and more stable. So the default is simple: if a check can be done a level lower, do it lower. Need to check a discount calculation — that’s a unit test, not a click through the whole checkout. The classics on this are Martin Fowler on the Test Pyramid and the practical test pyramid; on why not to dump everything into E2E — the Google Testing Blog.
A flaky test is worse than a missing one
This is probably the main thing. A test that fails every other run isn’t just useless — it’s toxic. It trains the team to ignore red. And once you turn a blind eye to one flaky, people stop trusting the whole suite, and a real bug slips by under the collective “eh, it always fails like that.”
The rule is simple: a flaky test you either fix or delete. “Mute and forget” is the worst option, because it leaves a hole in coverage and quietly lies that everything is covered. Well covered by Fowler on non-deterministic tests: a non-deterministic test loses a test’s main property — trust.
How not to argue with the team
To keep “automate or not” from turning into taste, I keep the criterion on paper, not “by feel.” Is it stable? Do we run it often? Is it expensive by hand? Is the result deterministic? Can it be checked a level lower? When the criterion is explicit, the “let’s automate this” argument is settled in a minute, not by emotion.
”Automate or not” checklist
- The check is stable (doesn’t change every sprint)
- Runs often (regression, every build), not once
- Expensive/slow/boring to repeat by hand
- The result is deterministic (no “by eye”)
- Can’t it be checked more cheaply a level lower (unit/API)
- Years of maintenance cost < the value
- The setup is proportional to the case’s value
- Ready to fix a flaky right away, not mute it
- Subjective/beauty/exploratory left to a human
In short
Automation isn’t “cover everything,” it’s a choice of where to invest an expensive resource that lives for years. A good automation engineer isn’t the one who writes more tests, but the one who knows which tests not to write. “Automate everything” sounds ambitious, but in practice it kills trust in the suite. Better fifty tests people trust than five hundred people hit retry on.
Sources: Martin Fowler — Test Pyramid, The Practical Test Pyramid, Google Testing Blog — Just say no to more E2E tests, Martin Fowler — Eradicating non-determinism in tests.