reliability
7 articles
-
Remote Config: How to Test What Changes Without a Release
A Remote Config hotfix didn't apply because an old cached config overrode the new one, and the 'fix' was verified on a build players don't have. A first-person breakdown: config versions, when changes actually reach the player, compatibility with old clients, and a QA checklist.
-
A retry is not a second operation: testing retries in QA
A practical plan for testing retries: idempotency, timeouts, duplicates, response ordering and recovery after a network interruption.
-
Ariane 5, Flight 501: How One Overflow Blew Up a Rocket in 40 Seconds
On 4 June 1996, the first Ariane 5 self-destructed 37 seconds after launch. The cause: an overflow converting a 64-bit float to a 16-bit int in reused Ariane 4 code. A first-person breakdown: what happened, why redundancy didn't help, and six QA lessons on code reuse, boundaries, dead functionality, error handling, and testing in the real configuration.
-
Testing Restore From Backup: The Backup Nobody Ever Restored
Everyone has backups — almost nobody tests the restore. A first-person take: why "backup completed successfully" guarantees nothing; what RTO and RPO are and why you measure them; what actually breaks during a restore (corrupt archive, wrong point in time, drifted schema, foreign environment, secrets); how to run a restore drill; and why you should monitor the restore, not the backup job. With a breakdown of the GitLab 2017 incident.
-
Rate limiting — how to test the limits everyone remembers only after an incident
While nobody is hammering the API, limits seem unnecessary — their absence is invisible right up until the first incident. A first-person take: the limit as a two-sided contract (the server restricts — the client survives it), the N/N+1 boundary and an honest 429 with Retry-After, key scope and how an account-based limit lets attackers DoS a victim, the burst at window boundaries, bypasses via X-Forwarded-For and sibling endpoints, the zones where a limit is mandatory (OTP, reset, promo codes), and why 'limits are off on staging' equals an untested production.
-
"It shows me old data" — how to test caches, the quietest source of bugs
The "user sees stale data" bug doesn't reproduce, gets closed as "went away on its own" — and comes back a week later. That's not mysticism, that's a cache. A first-person take: the map of six caching layers (browser, CDN, gateway, application, database, mobile client), invalidation as the main test case, caches leaking other users' data, cache stampede after a deploy, the "every case twice — cold and warm" rule, and why testing with the cache off means testing a system that doesn't exist.
-
Chaos Engineering for QA: deliberately breaking the system to test resilience
Resilience you never tested with a deliberate failure is an assumption, not a fact. Chaos Engineering for QA: the steady-state hypothesis, blast radius and the abort button, what failures to inject (instance kill, latency, dependency outage, resource exhaustion, zone outage), tools (Chaos Monkey, Gremlin, Chaos Mesh, AWS FIS, Toxiproxy), Game Days, the QA role (graceful degradation, retries, circuit breakers, observability) and a safe-experiment checklist.