mobilegame-qaapp-updatesmigrationsreleaseqa

Update testing — the bugs only users on old versions ever see

There’s a special breed of release bugs: “my progress disappeared after the update,” “the update logged me out,” “updated — now it crashes on launch.” They share one trait — on the test environment they don’t exist. Our environment is clean: fresh install, fresh database, fresh login. But the user doesn’t get the release as a clean install — they get it as an update on top of a version they’ve been living in for six months, with all the accumulated data. Which means we tested something other than what actually ships to people.

Ever since I stepped on this rake a couple of times, the update itself has been a first-class test object for me — no less important than the feature. Here’s how I test it.

An update is new code reading old data

A clean install and an update differ in one thing: with an update, the new code inherits the old version’s entire household. The local database (SQLite/Room/Realm/Core Data), SharedPreferences/UserDefaults, serialized objects, cached files, tokens in Keychain/Keystore. All of it was written by the old version — in the old format.

If the database schema changed, you need a migration. If the serialization format changed — the old blob must either be readable or deliberately discarded. If a settings key was renamed — the old value should move over, not vanish. Every “we changed the storage format” in the developers’ release notes is a red flag for QA: this is where “my progress disappeared” lives.

The version matrix: users don’t update from N-1

It’s convenient to assume users update from the previous version to the new one. In reality the store has a long tail: someone is sitting on a six-month-old version with auto-update off — and jumps five releases at once. For them, it’s not one migration that runs but a chain: all the intermediate ones, in order. And if the version 3→4 migration assumed 2→3 had already run — and this user’s path is custom — you get a crash that reproduces only for them.

So my minimal update-test set is: previous version → new (the bulk of users), oldest supported version → new (the worst-case migration chain), and at least one “from the middle.” From this follows an organizational requirement people forget: an archive of old builds must exist. If the team can’t install a three-month-old version, it has nothing to test updates with.

What it looks like hands-on

The scenario is boring, and that’s exactly why it catches a whole class of bugs: install the old build, live in it — log in, accumulate progress, change settings, put something in the cart — and install the new version on top (adb install -r on Android, install-over via TestFlight/Xcode on iOS). After the update I go down the list: login survived (not logged out), progress/data intact, settings not reset, the app launches at all. Plus one pass over the core scenarios — not on a clean database, but on this “lived-in” one.

A separate trap: installing via the IDE or TestFlight isn’t always identical to a store update (signatures, split APKs, downloadable content), so the final check of a critical release should use the closest-to-store path available — an internal testing track in Google Play / a TestFlight build of the same configuration.

An update in the middle of life

The store updates the app in the background — the user never asked. Which means an update can land between sessions with unsaved state: the game minimized mid-battle, a half-filled form, a file mid-upload. Coming back after the update — what does the user see? Restored state, a sane scenario restart, or everything lost?

There’s no perfect answer here, but the behavior should be deliberate, not accidental. The test is simple: background the app in an “interesting” state, update, come back.

Force update: the least-tested screen in the app

Sooner or later a force update appears: “this version is no longer supported, please update.” On Android there’s the in-app updates API with two modes — flexible (the update downloads in the background while the user keeps working) and immediate (a blocking full-screen flow). What I check in the mechanism itself: the old version actually gets blocked when force is on, the “update” button leads to the store to the app’s page (not into the void), the dialog cannot be bypassed — minimize/kill/rotate/turn off the network and keep using the app, and separately — what happens when the store is unreachable or the update hasn’t rolled out to the user’s country yet: a dead end forever, or a sane message?

And the reverse case that hurts: force update gets enabled via server config before the new version is actually available in the user’s store. The app demands an update — the store says “you have the latest version.” The user is locked out. This scenario is worth checking every single time force is switched on.

Staged rollout: two versions live at once

Releases roll out gradually — staged rollout in Google Play, phased release in the App Store. Two things follow for QA. First — the rollout can be halted, and monitoring crashes/metrics of the new version in the first hours is part of the release process, not an option. Second, less obvious: while the rollout is in progress, the old and new versions live in production in parallel — for days, and with the long tail of non-updaters, for months. So the server and API must support both: a new backend can’t silently change a contract the old client reads. Every server-side change gets checked on the “old client + new server” pair, not just on the fresh combination.

Downgrade: officially impossible, happens anyway

Stores don’t let users roll back to an old version, but it happens regardless: testers and sideloading on Android, restoring an old device from a backup, rolling back a build in internal tracks. And then old code reads new data — nobody wrote the “down” migrations, and it’s a near-guaranteed crash loop: the app dies on launch forever, cured only by a reinstall with data loss. The minimum worth knowing about your product: what exactly happens on downgrade — and if the answer is “crash loop,” let it at least be documented known behavior, with the team’s testing instructions explicitly forbidding rollback on top of data.

First launch after an update is not a first launch

A class of bugs on the boundary with onboarding: after an update, the app decides the user is new — and shows the FTUE again, resets the “tutorial completed” flag, re-asks for permissions or shows the language picker. The cause is usually the same: the flag lived somewhere the update touched (or the key got renamed in the new version). The check belongs in the basic pass: update and confirm the app remembers — we’ve been here before. And the what’s-new screen, if there is one: shows once, closes, doesn’t reappear on every launch.

Antipatterns

Testing only the clean install. The big one. A clean install is a minority of a release’s real traffic; the bulk is updates on top.

Migrations without tests on real old data. A unit test of a migration on a synthetic database is good, but a real user database from an old version (the volume! the junk! collisions from even older versions!) regularly finds what synthetic data doesn’t.

Not keeping old builds. Without an archive of builds and old data, update testing is physically impossible. Set up a shelf of old versions — and snapshots of “lived-in” data to go with them.

Update testing checklist

  • Update from the previous version: login, data, settings, progress intact
  • Update from the oldest supported version (the migration chain)
  • Core scenarios on a “lived-in” database, not a clean one
  • Update with unsaved state: behavior is deliberate
  • Force update: can’t be bypassed, leads to the store, sane when the store is unreachable
  • Force isn’t enabled before the version is actually available to users
  • Old client + new server: the API contract supports both versions
  • Crash monitoring of the new version in the first rollout hours, halt plan
  • Downgrade: behavior known and documented
  • No repeated FTUE/permission prompts after the update, flags survived
  • An archive of old builds and data snapshots exists

A release isn’t “the new version works.” A release is “every user’s path from their old version to the new one works.” A clean install verifies the first. Update testing verifies the second — and that’s where the bugs live that everyone sees except us.