remote-configfeature-flagsmobilereliabilityqa

Remote Config: How to Test What Changes Without a Release

A tester writes: “The Mirror is nonsense. 3k damage out of nowhere, zero counterplay.” He means an enemy in my idle game that reflects part of your damage back at the tower. At difficulty tier 10 it killed in a single reflection.

The good news: every enemy number lives in Remote Config, so it can change without a release. The bad news: in the next hour I stepped on almost every rake remote configs have. This post is about them.

Why Remote Config is its own risk area

Remote Config (Firebase, LaunchDarkly, your own JSON from a server, it doesn’t matter) is code that reaches production around everything that normally protects production: no code review, no build, no regression run, no store review. You hit “Publish”, and a few minutes later live players get different behavior.

So the question “what to test” is not about the JSON itself, but about four things:

  • will the change reach the player, and when;
  • which client version it reaches and how that version reads it;
  • what happens if the config is broken or incomplete;
  • can this config actually fix the problem, or won’t numbers help.

Rake 1: an old config overrides a new one

The usual setup: the build ships with a default config, and on launch the app downloads the remote one and caches it. The cache wins over the bundled config.

What broke for me: I shipped a build with new enemies and a new bundled config. But players still had last version’s remote config in their cache, without the new enemies. It “won” and silently overrode the new one. The update shipped, and nothing changed in the game.

How to catch it:

  • every config has a version field;
  • client rule: a remote config applies only if its version is not lower than the bundled one;
  • test: install the old build, let it cache the remote config, update the app on top, and check which values are actually in the game.

Almost nobody runs that last test, because QA usually installs builds clean.

Rake 2: when changes actually reach the player

I published the fix, the tester restarted the game, and the Mirror still killed him. There were two reasons, both normal:

  • Applied on the next launch. The fresh config downloads in the background while the game is already running and applies on the next start. The first restart only downloads it, the second applies it.
  • Fetch throttling. The client doesn’t fetch on every start: Firebase has a minimum interval between fetches, and in release it’s usually set from minutes to hours. Firebase’s default is 12 hours.

For QA this means one simple thing: a ticket that says “fixed via config” must state how long and after which actions the player sees the change. Otherwise the tester rightly says “not fixed”.

In the end I changed it: the config applies as soon as it arrives, unless the player is in a battle. If they are, it applies right after the run ends. Changing balance mid-fight is risky: enemies on the field point to types from the old config.

Rake 3: verify the hotfix on the build players have

This was the main lesson of the day. I fixed the code (added a damage cap for the enemy), tuned the numbers, ran a check on the phone, and everything looked great. Except I checked on the new build, which already had the cap. The tester had the old store build without the cap, and there the only thing that can help is the Remote Config numbers.

The right way: build exactly the commit that is in the store now, install it on a device, pull the fresh remote config, and reproduce the player’s scenario. That’s what I did after I got caught. With the old numbers: “killed by Mirror” on wave 12, exactly like the tester. With the hotfix: the Mirror kills nobody.

The same rule covers compatibility: an old build simply ignores new fields in the config. So a hotfix for old clients can use only the fields the old code already reads.

Rake 4: numbers don’t always save you

The Mirror’s reflection strength depended on the enemy’s own attack, and attack grows with the difficulty tier. I could tweak coefficients all I wanted, but at high tiers the formula still ended in “one-shot”. It needed a cap based on the tower’s health, the formula didn’t have one, and a config can’t add it.

What to test in advance:

  • every damage and reward formula has a cap that doesn’t grow with the level, or that’s a conscious decision;
  • for every config parameter it’s clear which range is safe;
  • invariants are checked by an automated test: “one hit from a regular enemy never takes more than X% of health”, “a wave’s reward never exceeds Y”.

Rake 5: a broken or incomplete config

Someone makes a typo in the JSON, deletes a field, or puts a string instead of a number. What does the client do?

A minimal set of checks:

  • invalid JSON: ignored, the bundled config stays, a warning in the logs;
  • missing field: the default value is used, not zero (zero “speed” or “damage” easily breaks a game);
  • wrong type: doesn’t crash;
  • empty key in the console: behaves the same as a missing key;
  • huge or negative parameter: the client doesn’t loop forever or divide by zero.

Rollback, targeting and gradual rollout

The most valuable thing about Remote Config is that you can roll it back. But that only works if:

  • version history is on and someone knows how to roll back (in Firebase it’s the template history);
  • changes roll out not to everyone at once: a percentage of users or a specific group, testers first;
  • targeting conditions are checked: app version, country, language. A common mistake is a “version ≥ X” condition with a typo, and the config reaches nobody at all.

A good general framework for kinds of flags and their lifecycle is Martin Fowler’s article on feature toggles.

Checklist

  • Every config has a version, a remote one never overrides a newer bundled one
  • Upgrade from an old build with a cached config is tested
  • It’s known when a change reaches the player: background download, interval, whether a restart is needed
  • The config doesn’t change critical logic mid-session: a battle, a payment, a save
  • The hotfix is verified on the store build, not on a branch
  • A hotfix for old clients uses only fields the old code reads
  • Damage and reward formulas have caps, parameters have safe ranges
  • Invalid JSON, a missing field and a wrong type don’t break the app
  • There is a rollback plan and a gradual rollout
  • Targeting conditions are checked on a real device

Remote Config saves releases, but it doesn’t remove testing. It just moves testing to where it’s easy to forget: the gap between what you published in the console and what is actually running on the player’s phone.