Testing Feature Flags and A/B Experiments: How Not to Ship a Mess of Variants

We once “rolled out” a harmless flag: a new onboarding screen for 10% of the audience. QA ran the new screen — works. A day later support blew up: some people saw the old onboarding one launch and the new one the next, losing their progress each time. The flag wasn’t “sticky”: on every launch the user was re-bucketed at random. We tested the screen, but we never tested the flag’s behaviour.

Feature flags and A/B are no longer “hide a button behind an if.” They’re a branch of code running straight in production — a whole layer that lives its own life. And you test not just the new feature, but the switching mechanism itself.

A flag isn’t “on/off” — it’s a branch of code in production

The moment if (flag.isOn("new_checkout")) appears, you have two versions of the app live in production at once. Both reach real users. So you test both — including the old branch, because it’s easy to break “on the way” while you build the new one.

The minimum matrix for a single boolean flag:

ON — the new feature works.
OFF — the old behaviour is untouched (regression!).
Toggle on the fly — what happens to a user sitting in the app the moment the flag flips: picks it up without a restart, breaks, or shows a half-assembled screen.
Flag unavailable — the flag service didn’t answer / timed out. The app must fall back to a default, not crash and not hang on a white screen.

That last one gets forgotten most often, and it’s the most dangerous: the flag network goes down → the whole app goes down.

What actually breaks (not what you meant to test)

Feature-flag bugs are almost never in the feature itself. They’re in the seams:

Flag combinations. Five boolean flags is 32 states. Nobody tests all of them, but two or three flags that touch the same screen must be checked together. The classic: flag A hides the old button, flag B relies on it — turn both on, the button’s gone, checkout can’t start.
Defaults. What does the SDK return when there’s no network / no user / the config hasn’t loaded yet? The default must be the safe old behaviour, and you verify it by hand with the network off.
Cache and evaluation timing. The flag was evaluated at launch and cached, and you flip it in the dashboard expecting an effect — but nothing happens until a restart/refetch. That’s not a bug, it’s design; but knowing when the flag is re-evaluated is critical for testing.
Races. The flag hasn’t arrived yet, but the code already asks for its value → the branch is chosen by default, then “jumps.” UI flicker, duplicate analytics events.

Targeting and segments — where the nasty bugs live

Once a flag isn’t “100% or 0%” but “10% / iOS only / premium only / Germany only,” you have targeting — and that’s where the fun hides.

Boundary segments. A user right on the edge: subscription expiring this very second, app version minVersion − 1, region by IP vs by settings. Check which bucket they land in and whether they “flicker” between them.
Sticky / stickiness. The same user must stably see one variant across sessions and devices. If bucketing isn’t tied to a stable key (user id, not session), the person will jump between variants — exactly the story above. Easy to test: kill the app, come back several times — the variant doesn’t change.
Percentage rollout. 10% is roughly 10% by hash, not “the first 10% in a list.” Verify that bumping 10% → 50% doesn’t throw those already in the feature back out (or you lose experience consistency).
Who wins. If two rules match a user (both “premium” and “Germany”), which one fires? Rule priority is its own test case.

An A/B experiment: test the mechanism, not the hypothesis

An important fork: a feature flag answers “should we show the feature,” an A/B experiment answers “which variant is better on a metric.” QA doesn’t test the hypothesis (“will conversion go up”) — that’s analytics’ job. QA verifies the experiment is built so the data can be trusted:

Exposure is counted once, at the right moment. The “user saw variant B” event must fire when they actually saw it, not when the config loaded. Otherwise people who never reached the screen enter the sample and blur the result.
The split converges. Meant to be 50/50 — in practice roughly 50/50 too. A strong skew = broken bucketing or double exposure.
Nobody lands in two variants at once. One user — one variant, forever, across all devices.
The metric is tied to the variant. The purchase event carries the right variant/experiment_id. Verify by intercepting traffic (Proxyman/Charles) — look at what’s actually sent, not what “should” be.
Mutually exclusive experiments don’t overlap when they’re not supposed to (mutual exclusion groups).

Kill switch: the main test is “turn it off”

Any risky flag must be instantly revertible from the dashboard, without a release. And you rehearse this scenario before an incident, not during one:

I turn the flag off in the dashboard → how fast does the effect reach the client (instant / next refetch / after a restart)?
Users who were in the feature return to the old behaviour cleanly, breaking nothing on the way (unfinished transactions, open screens)?
Analytics stops sending the exposure event.

If “turning it off” actually needs a release, it’s not a kill switch, and you haven’t removed the release risk.

Zombie flags

Every flag is tech debt and an extra branch in the matrix. A flag that’s “already at 100% and has lived forever” must be deleted along with the dead code branch. While it lingers:

someone accidentally turns off long-default behaviour;
the old (OFF) branch quietly rots untested and surfaces one day;
the state matrix grows for nothing.

It helps for QA to keep a list of active flags and ask at retro: “is this still an experiment or is it cleanup time?” GrowthBook and LaunchDarkly can surface stale flags — use it.

Pre-release checklist for a flag

Both branches tested: ON and OFF (OFF = regression of the old behaviour).
Safe default when the flag service is unavailable (verified with the network off).
It’s known when the flag is re-evaluated (launch / refetch / restart) — and it’s verified.
Stickiness: the variant is stable across sessions and devices.
Combinations with other flags touching the same screen are checked.
Targeting: boundary segments, rule priority, correctness of the percentage rollout.
For A/B: exposure counted once at the right moment; the split converges; variant in events (verified by intercepting traffic).
The kill switch truly turns it off without a release — rehearsed.
A ticket exists to delete the flag after rollout.

In short — what to take with you

A flag is two versions of the app in production at once. Test both branches, not just the new feature.
The most forgotten and nastiest case is the flag service being unavailable: there must be a safe default, not a white screen.
Stickiness and “when the flag is re-evaluated” matter more than the feature itself. Verify by restarting.
QA tests A/B as a mechanism: exposure once, split converges, variant in events — the data must be trustworthy.
Rehearse the release-free kill switch ahead of time. Delete zombie flags.

Further reading: GrowthBook — feature flags · LaunchDarkly — testing feature flags · Martin Fowler — Feature Toggles