reliabilitybackupdisaster-recoverynonfunctionalqa

Testing Restore From Backup: The Backup Nobody Ever Restored

On 31 January 2017, a GitLab engineer, mid-incident, accidentally deleted the directory with the production database — 300 GB gone in front of them. Then it got interesting: of the five backup and replication mechanisms they had, not one worked — the dumps turned out to be a couple of kilobytes, replication lagged, snapshots were in the wrong place. What saved the company was a chance copy taken by hand 6 hours earlier for an unrelated task. Their public postmortem teaches the subject better than any article.

The lesson to internalize before it happens to you: a backup nobody has ever restored from isn’t a backup, it’s a hope. And testing that hope is QA’s job.

A backup is not a restore

“Backup completed successfully” means exactly one thing: the write process finished without an error. It does not mean:

  • the archive contains data (rather than an empty file from a mysqldump that lacked permissions);
  • the archive isn’t corrupt and will unpack;
  • you can bring a working system up from it;
  • the data inside is consistent (the snapshot was taken while half the transactions were mid-flight).

The only proof a backup is alive is a successful restore from it. Everything else is a green checkmark you’d like to trust.

RTO and RPO — two numbers without which a backup is meaningless

Before testing anything, agree on two numbers:

  • RPO (Recovery Point Objective) — how much data you can afford to lose, measured in time. A daily backup = RPO of 24 hours: in a disaster you lose up to a day. Fine for a blog, a catastrophe for payments.
  • RTO (Recovery Time Objective) — how fast the system must be working again. If restoring 500 GB takes 9 hours and the business assumed an RTO of 1 hour, you have a mismatch — and learning about it during the disaster is the worst way.

QA doesn’t verify “is there a backup,” it verifies whether a real restore fits the promised RTO/RPO. That’s measurable — so measure it.

What actually breaks during a restore

Failures are almost never in the restore command itself. They’re in the seams:

  • Empty or corrupt archive. The dump ran, but with a permission/space error — garbage inside. Only caught by unpacking and checking a checksum.
  • Wrong point in time. You need to restore to “5 minutes before the incident,” but all you have is yesterday’s full dump — point-in-time recovery was never set up.
  • Drifted schema. The backup is of the old DB version, the code has already moved to a new schema — the restore exists, but the app won’t start on it.
  • Dependencies and secrets. You restored the DB but lost the encryption keys / environment variables / external configs — the data’s there, access isn’t.
  • Foreign environment. The backup was taken on one Postgres version/hardware and is being brought up on another — incompatibility surfaces at the worst moment.
  • Order and volume. What to restore first so the service comes up; and whether there’s enough space/time for the full volume under pressure.

Restore drill: a rehearsal, not a hope

A restore drill is a planned exercise: take a real backup and fully bring it up in an isolated environment, as if prod just died. What you check:

  • The restore runs to completion, with no manual “oh, and we’ll also patch this here.”
  • The data is consistent: counters, sums, relationships between tables reconcile against a reference (not just “the files are there”).
  • The application on the restored database actually starts and works — login, a key flow, checkout.
  • You time it: did you fit the RTO or not.
  • You check the latest backup and restore to different points (yesterday, a week ago) — rotation and older copies must be alive too.

Do it regularly and, where possible, automatically — a quarterly manual drill gets forgotten, while an automated drill in CI catches “it broke silently” immediately.

Monitor the restore, not “the backup job passed”

The classic trap: the alert fires on “the backup job failed.” The job didn’t fail — it honestly wrote 2 KB of nothing. No alert, everyone’s calm, no data.

What to actually monitor:

  • the size of the backup and its deviation from yesterday’s (a sharp drop = alarm);
  • the age of the last successful backup (has it aged past the RPO);
  • the result of the last restore drill — green only if a working system actually came up from the backup, not if the write job returned 0.

Checklist

  • RTO and RPO are defined and agreed with the business for each critical system.
  • There’s a restore drill: the backup is regularly brought up in an isolated environment to a working state.
  • Data consistency is checked after the restore, not just file presence.
  • The application starts on the restored DB and passes a key flow.
  • The restore time is measured and checked against the RTO.
  • Point-in-time recovery is tested (not only “yesterday’s full dump”).
  • Old/rotational copies are restored too, not only the latest.
  • Secrets, encryption keys, configs are accounted for — without them the data is useless.
  • Monitoring watches size/age/drill result, not “the job didn’t fail.”
  • There’s a runbook an on-call can follow at 4 a.m., not only the author.

In short — what to take with you

  • “Backup completed” only proves the write. The only proof a backup is alive is restoring from it.
  • Without RTO/RPO you can’t test a backup — “success” is undefined.
  • What breaks isn’t the restore command, it’s the seams: corrupt archive, wrong point in time, schema, secrets, foreign environment.
  • A restore drill — regular and ideally automated — is the only way to know instead of hope.
  • Monitor size/age/restore result, not the job’s green checkmark.

Further reading: GitLab — postmortem of the January 31, 2017 incident · Google SRE Book — Data Integrity