non-functionalcachingreliabilityperformanceqa

"It shows me old data" — how to test caches, the quietest source of bugs

A familiar ticket: “The user sees outdated data.” You go check — everything’s fresh on your end. You write “cannot reproduce,” the user replies “it fixed itself.” Closed. A week later, the same ticket from another user. Then another one. And this is where it’s worth stopping, because “it fixed itself” is almost never mysticism or “a glitch.” It’s a cache: the data was updated, but one of its copies wasn’t — and that copy died exactly on its TTL, which is why it “fixed itself.”

There’s a classic joke that there are only two hard things in computer science: cache invalidation and naming things. It’s funny because it’s true. And yet caches almost never get tested deliberately — they’re just “somewhere out there” until they start showing stale data. Here’s how I test caches and why I consider them the quietest source of bugs.

Why cache bugs are special

A regular bug reproduces with steps. A cache bug reproduces with steps and time: the result depends on when you look and who hit that endpoint before you. The first request warms the cache, the second reads from it — those are two different code paths, and the bug may live in only one of them. On top of that, on a clean staging environment where nobody but you goes, half of the cache effects simply never show up.

Hence the main reason such bugs get closed as “cannot reproduce”: the tester arrives after the TTL, the cache has already died, everything is fresh. The user came earlier — and saw the stale copy.

First, the map: where caches actually live

You can’t test a cache blindly — you need to know where it is. There are usually more layers than you’d think:

  • browser — the HTTP cache driven by Cache-Control/ETag headers;
  • CDN / edge — Cloudflare, CloudFront and friends, caching responses close to the user;
  • API gateway / reverse proxy — nginx, Varnish, route-level caching;
  • application — in-memory or Redis/Memcached: query results, sessions, feature flags;
  • database — query caches, materialized views;
  • mobile client — its own layer on top of everything: the SDK’s HTTP cache, image caches, a local database synced “eventually.”

A “shows old data” ticket can live on any of the six. So the first question when testing a feature isn’t “is there a cache” but “on which layers and with what TTL.” If the team can’t answer — that’s already a finding.

Invalidation: the main test case

Testing a cache “on read” is boring — it either works or it doesn’t. Everything interesting starts on data changes: you changed something → where did it update, and where is a stale copy still sitting?

The buggiest scenario is changing through one channel and reading through another. The price was updated through the admin panel — but the mobile app reads through an API with a cache that knows nothing about the admin panel. The user edited their profile on the website — the app shows the old name for another hour. These combinations should be written out explicitly: every write path × every read path, and for each pair ask “who invalidates the cache, and how.”

And separately — the TTL boundary: what does the user see at the moment the cache has expired but the fresh copy isn’t warmed yet? An error? A slow response? The stale data once more?

Caches and other people’s data — the most expensive class of bugs

Functional cache bugs are unpleasant, but there’s a worse class: the cache serving one user’s data to another. It happens in mundane ways: a response was cached without regard to whom it belongs — the cache key doesn’t include the user id, or the CDN cached a personalized response as a shared one, or someone forgot the Vary header and a user with a different language/currency/permissions receives someone else’s version of the page.

What I always check: log in as two different users with different permissions and data — and see whether the first user’s response “leaks” to the second. And the reverse case: the user logged out / had their permissions revoked — is the cached data still alive and accessible? A cache that survives logout and permission changes isn’t a performance concern anymore — it’s a security one.

Cache stampede: the cache as a load-testing case

While the cache is warm, it hides the real cost of requests. Now comes a deploy — and the cache is flushed. All the requests that were served from cache for months hit the database simultaneously. If the database isn’t ready for that — production collapses exactly at release time, and in the logs it looks like “everything died after the deploy,” even though the code is innocent.

This is called a cache stampede, and it’s a testable scenario: how does the system behave with a cold cache under normal load? Run your load test twice — with a warmed cache and with a flushed one — and compare. The difference is your risk on every deploy. A related question belongs here too: if the data source is down, does the cache serve a stale copy or an error? The stale-while-revalidate pattern was described back in RFC 5861 — but it only works if someone actually verified its behavior.

Caches and releases

A cache lives longer than code. After a deploy, the cache holds objects serialized by the old version: if the format changed — the new code reads the old cache and crashes (or silently returns mangled data, which is worse). The flip side: a client with a cached old bundle talks to the new API. Questions for every release: what in the cache will survive the deploy, can the new code read it, and does the cache need a manual flush (and is that written in the release checklist)?

What to look with

The tooling is minimal and simple. For HTTP layers — DevTools (the Network tab: where the response came from, from disk cache / from memory cache), curl with headers, and Proxyman/Charles for the mobile client: you can see Cache-Control, ETag, Age and CDN headers like cf-cache-status: HIT/MISS — they tell you which layer answered. For server-side caches — Redis CLI: inspect a key, its TTL (TTL key), delete it and verify the system survives the miss. If personalized data has cache keys without a user id — there’s your bug from the “other people’s data” section, found in a minute.

Antipatterns

Testing with the cache turned off, “so it doesn’t get in the way.” I get the urge — results are unstable with a cache. But production runs with the cache on, so a stage without it tests a system that doesn’t exist. The cache isn’t an obstacle to testing — it’s a subject of it.

Fixing things with “clear the cache” without asking “why.” If a bug is cured by clearing the cache, that’s not a solution — it’s a diagnosis: invalidation is broken. The next user has the same bug, and nobody will clear the cache for them.

A stage without caches while production has them. A variation of the first one at the infrastructure level: no CDN/Redis on staging “for simplicity” — and the entire class of cache bugs ships to production unverified.

Cache testing checklist

  • A map of layers: where the caches are, what TTL, who invalidates
  • Every case twice: cold cache and warm — they’re different code paths
  • A change through every write channel → verified through every read channel
  • Two users with different permissions/data: nothing leaks
  • Logout and permission changes: cached data doesn’t survive them
  • Vary/cache keys account for language, currency, platform, user
  • Cold cache under load: a deploy doesn’t take down the database
  • Source is down: stale or error — the behavior is deliberate, not accidental
  • Release: old cache readable by new code, flushing is in the release checklist
  • “Cannot reproduce” + “fixed itself” = check the TTL before closing

A cache is a deliberate decision to show the user slightly stale data in exchange for speed. A perfectly fine decision. What’s not fine is when nobody can say how stale, to whom, and what happens when the copy and the original diverge. That’s the tester’s job: to make the cache’s trade-off visible and verified instead of accidental.