securitydistributed-systemspaymentsapitesting

Idempotency and retry storms — what QA must test in distributed systems

The most expensive class of bugs in a payment system isn’t “didn’t go through” — it’s “went through twice”. A user double-clicked Pay, or a client timeout caused a retry, or your queue delivered a message more than once — and the card got charged twice. That’s money, support tickets, chargebacks. Every such problem is the consequence of untested idempotency.

This guide is what idempotency means from a QA’s chair, the 5 typical retry/duplicate scenarios you must test, what a retry storm is and how to reproduce it, concrete tools (WireMock, Toxiproxy, k6), and a checklist before shipping a payment or webhook endpoint.

What idempotency is

An operation is idempotent if a repeated call with the same parameters yields the same result and creates no side effects. Easy example: GET /api/user/123 — idempotent (read-only, repeats don’t change state). Harder example: POST /api/charge {amount: 100, idempotency_key: 'abc123'} — idempotent ONLY if the server stores the key and on a repeat returns the cached response instead of charging again.

HTTP methods, per RFC 7231:

  • GET, HEAD, OPTIONS — always idempotent (read-only).
  • PUT, DELETE — idempotent by definition (replace/remove a resource, repeats land in the same state).
  • POST — NOT idempotent by default. Add it manually via Idempotency-Key.
  • PATCH — depends on semantics (if patch is a relative change like balance += 10, not idempotent).

The most common mistake — POSTs without idempotency on payments, forms, orders. Stripe’s Idempotency-Key is the canonical reference.

Idempotency-Key: how it works, how to test

The client generates a unique key (UUID v4) and sends it in the Idempotency-Key header on a POST. On first request the server performs the operation and stores {key → response} in Redis/DB for 24 hours. On a repeat with the same key — it returns the stored response without re-executing.

What QA tests:

  • First POST with key → creates the resource, returns 201/200.
  • Repeated POST with the same key → returns 200 with the same body, resource is NOT duplicated.
  • Same key with a DIFFERENT body → must return 422 or 409 (key conflict), not silently execute a new request.
  • After the 24-hour TTL the key can be reused without collision.
  • Parallel concurrent requests with the same key (race condition) — must yield one result, not two.
  • What happens if the key is empty, missing, or too long (Stripe caps at 255 chars)?
  • What happens if the key exists but the server crashed between performing the operation and writing the idempotency record?

The last case is the most interesting. If the process crashes after the operation but before the key is persisted, the retry creates a duplicate. The fix is atomicity via a transaction or an idempotency-table-first pattern.

5 typical retry/duplicate scenarios

1. Double-submit form

User clicks Pay → nothing visible happens for 2 seconds (just slow network) → clicks again. The browser sends two POSTs. Without idempotency — two charges.

How to test: Postman runner with two identical requests fired near-simultaneously. Disabling the button in the UI after click is UX, not security. Server-side idempotency is the security boundary.

2. Client timeout retry

Client sent POST → waited 30 seconds → timeout → resent. But the first request arrived and was processed! The server returned 200 but the client never saw it. Now there are two charges.

How to test: Toxiproxy (github.com/Shopify/toxiproxy) — add 60 s latency to the endpoint, run the client with retry-on-timeout. You must observe exactly one charge.

3. Third-party webhook retry

Stripe / Twilio / GitHub send a webhook → your endpoint returned 500 → the provider retries after 1, 5, 30 minutes, then 1 hour, up to 3 days. If the backend actually processed the first request but didn’t return 200 in time — all retries will reprocess.

How to test: WireMock simulating slow responses. Or just send the same payload (with the same event id) twice — must be processed once. On the backend side, store event_id → processed_at and check before processing.

4. Queue redelivery

Kafka, SQS, RabbitMQ — all deliver messages at-least-once by default. If a consumer crashes after commit but before send_response — the message is redelivered. If the consumer acks but the backend didn’t persist the result — same story.

How to test: kill the consumer mid-processing (SIGKILL during processing). Bring it back. Check there’s exactly one result in the DB.

5. Cron job restart

A cron started “daily reconciliation” → process crashed after 10 minutes → cron sees the crash and restarts (if configured to). Or a Kubernetes pod retry. Without idempotency — users get duplicate reports, duplicate summaries, duplicate credits.

How to test: run the cron logic twice in a row. The result must be one. Inside the cron — store a “processed_at” marker on the resource.

Retry storms — how one service breaks its neighbors

Service A depends on service B. B is down for 30 seconds. A receives 503 → retries every 100 ms. At peak, A is firing 100 retries per second at B. When B comes back — it gets a storm of 30000 queued retries, can’t keep up, crashes again. This is cascading failure.

The canonical write-up from AWS: Timeouts, retries, and backoff with jitter. The solution is four techniques QA must see and test:

Exponential backoff with jitter

Retry not every 100 ms but at 1, 2, 4, 8, 16, 32 seconds. Plus a random addition (jitter) between 0 and that value. This spreads retries over time, avoiding a thundering herd.

Retry budget

The client has a per-minute/per-hour retry cap. If the service is down for 10 minutes — after 10 retries the client gives up and surfaces the error to the user. Not nicer, but it protects against the storm.

Circuit breaker

See Martin Fowler: CircuitBreaker. The client tracks the error rate on a downstream. Above a threshold it “opens” the circuit and stops trying for 30 seconds (no retries), then “half-opens” — probes a single request. Success closes it; failure re-blocks.

Retry-After header

On 429 (rate limit) or 503, the server can return Retry-After: 60 — the client MUST wait that many seconds before retrying. QA checks: does the client respect that header (and not just ignore it)?

What QA tests in retry logic

  • Client retries on 5xx, does not retry on 4xx (4xx is a client error; retrying is pointless).
  • Client does NOT retry on 400, 401, 403, 422 (no help from retrying).
  • Client retries on 429 but respects Retry-After.
  • Exponential backoff actually grows (not a fixed 100 ms).
  • Jitter is applied (two parallel clients retry at DIFFERENT moments).
  • Max retries is capped (e.g. 5), not infinite.
  • Circuit breaker opens after N consecutive 503s and stops requests.
  • Test with varying latency — Toxiproxy lets you inject delay 100 ms, 1 s, 10 s, dropout, slicer.
  • Simulate a retry storm: 100 parallel clients, backend down — after recovery the backend must NOT die from the burst.

Tools

WireMock — fault injection

github.com/wiremock/wiremock — an HTTP mock that you can configure to “return 503 the first three times, then 200”, “return random between 200 and 500”, “respond after 30 seconds”. A stand-in for a downstream service in retry tests.

Toxiproxy — latency, dropout, slice

github.com/Shopify/toxiproxy from Shopify. TCP proxy with injection: add delay, drop packets, slow read, timeout emulation. Between the client and the real server — for integration tests, not unit.

k6 — load + retry storms

k6.io/docs. A JS script generating 100-1000 virtual users; retry logic configurable. Useful for reproducing a retry storm — fire 100 VUs that retry on 503 and watch backend metrics.

Postman / Newman runner

For functional idempotency checks (same key → same response). Pre-request script to generate the UUID, test script to compare the two responses.

Checklist before shipping a payment / webhook endpoint

  • POST with an Idempotency-Key returns the same response on repeat with the same key.
  • Same key with a DIFFERENT body returns 422/409, doesn’t silently execute a new request.
  • Idempotency-key TTL is documented and tested (typically 24 hours).
  • Parallel requests with the same key → one result (race condition closed).
  • The webhook endpoint processes one event_id exactly once, even on redelivery.
  • Client retry logic: 5xx — retries, 4xx (except 429) — doesn’t retry.
  • Client respects the Retry-After header.
  • Exponential backoff with jitter, max 5 retries.
  • Circuit breaker opens after N consecutive failures.
  • Retry storm test: 100 parallel clients, backend down 1 minute, recovers — backend doesn’t die from the burst.
  • Queue consumer: kill mid-processing → exactly one result in the DB.
  • Cron / scheduled job: two consecutive runs → identical outcome to one run.
  • CI integration tests with WireMock fault injection + Toxiproxy latency.

Idempotency isn’t a “hard problem”, it’s a “discipline”. 90% of duplicate-charge / duplicate-webhook / duplicate-order bugs are caught by a single unit test that sends the same request twice and asserts a single effect. Retry storms are caught by a single integration test with WireMock. Teams that skip this — sooner or later pay in chargebacks and lost trust for invisible bugs that were the easiest to catch.