apireliabilityqadistributed-systems

A retry is not a second operation: testing retries in QA

The payment button was tapped once, but the order was created twice. The user saw a timeout and tapped again. The client retried because it never received the response. The server had already completed the operation, but the response was lost on the way back.

For QA, “test retries” is not one scenario. You need to know what can be retried, who performs the retry, and how the system distinguishes a new operation from a continuation of the old one.

Start with the contract

For every request, record whether it may be retried, who retries it, the maximum attempts, which errors are temporary, and what the user sees after retries are exhausted.

GET is usually safe to repeat, but even a read can trigger a side effect in a poorly designed endpoint. POST does not become safe to repeat just because the client added a timeout. Creating an order, charging a payment, or granting a reward needs an explicit idempotency mechanism.

The expected result should describe server state, not only an HTTP status. Repeating the same operation with the same key should leave one operation, and the client should receive a consistent result for it.

Separate the reasons for retrying

Test a network failure before sending, a failure after sending, a response timeout, temporary 502/503 responses, 429 rate limits, and explicit business errors separately. Retrying every error is dangerous: another attempt will not fix a wrong password or insufficient funds.

For each cause, check the actual request count, delays, and headers. Retries may happen in the SDK, HTTP client, proxy, and queue. Three configured attempts can become twelve.

Inject a controlled delay on a test server. It helps reproduce the case where the server has changed state while the client still considers the request unsuccessful.

Test idempotency

Send the same operation several times with one idempotency key. Verify that one entity is created, one charge occurs, and repeated responses point to the same result.

Then reuse the key with a changed body. The server should return the original result or explicitly reject the conflict. Silently executing a second operation creates a difficult defect.

Check empty and overlong keys, casing rules, and key retention if those constraints are part of the contract. Document what happens after retention expires.

Check ordering and UI state

During a retry, the user must not see two conflicting states such as “payment complete” and “failed”. Check button locking, progress state, screen reopening, and recovery after an app restart.

Deliver responses out of order: a late first-attempt response arrives after the retry succeeded. The UI must not roll back confirmed state using an old response.

For offline queues, verify that one operation is not sent twice after connectivity returns. Switching accounts before the queue is sent deserves its own case: an event created by user A must not be sent as user B.

Check observability

Logs and metrics should distinguish request_id, operation_id, and attempt number. A correlation_id alone is insufficient if it cannot show whether three records are retries of one operation or three separate operations.

Alerts should reflect the final operation result, not only the number of HTTP responses. Five temporary failures followed by one success are a different signal from five created orders.

The success criterion is simple: repeating a request does not change the operation unexpectedly, and the team can reconstruct the attempts and their outcomes from the evidence.