non-functionalperformanceload-testingobservabilitytesting

Load testing: how to design it and not get garbage numbers

“We ran 1000 threads, average 200 ms — fine, ship it.” That’s a load test that proved nothing. A good load test answers two questions: will the system handle the expected load and where exactly does it break. For the numbers not to be garbage, the test has to be designed. Let’s break it down.

Types of load tests — what each finds

  • Load — the expected load (target RPS): do we hold the SLA in production.
  • Stress — above expected, to failure: where and how it breaks, does it recover.
  • Soak / endurance — normal load, but for hours: memory/connection leaks, degradation, log/disk overflow.
  • Spike — a sudden surge (sale, push campaign): how we survive the jump and the drop.
  • Breakpoint / capacity — slowly raise the load until it breaks: find the real ceiling.

Metrics: percentiles, not the average

The main self-deception is the average latency. An average of 200 ms can hide that every twentieth request is 3 seconds. Look at percentiles: p95, p99, p99.9. It’s the tail that users feel. What to measure:

  • Throughput — actual RPS (not the number of virtual users).
  • Latency by percentiles — p50/p95/p99/p99.9; the average is for the bin.
  • Error rate — the share of errors and timeouts under load (a green test with 30% 5xx is not “passed”).
  • Saturation — resource saturation (CPU, memory, IO, network, pools) on the system side.

Open vs closed model — where lies happen most

This decides how realistic the test is:

  • Closed model: a fixed number of virtual users, each sends the next request only after the response to the previous one. If the system slows down — the load drops by itself. Result: you mask the problem — the queue doesn’t grow because you don’t create it.
  • Open model: a fixed arrival rate of requests (N per second) regardless of whether the system responded. That’s how real users behave and queues start to grow — you see the real degradation. Details — in the k6 docs on open vs closed.

If you’re measuring a service’s throughput — you almost always need the open model. The closed one understates problems under load.

Realism of the load

  • Ramp-up and warm-up: “instantly 1000 users” is unrealistic and catches cold caches/JIT/pools. Ramp up gradually, let the system warm up, measure on the stable section.
  • Traffic profile: don’t hammer one endpoint. Reproduce the real mix (login, listing, search, payment) in real proportions.
  • Think time: real users think between actions — build in pauses (in the open model — via arrival rate).
  • Data: not an empty DB. On an empty table any query is fast; under real volume indexes, locks kick in, the query plan changes.

Measure ON THE SYSTEM SIDE, not just the client

Client numbers (latency, RPS) tell you what is slow, but not why. To find the bottleneck, capture server-side metrics — essentially the golden signals (latency, traffic, errors, saturation, see Google SRE):

  • Saturation of CPU/memory/disk/network on each service.
  • The DB connection pool and wait time in the pool (a common hidden ceiling).
  • Queue length (brokers, thread pools), GC pauses, GC time.
  • DB metrics: slow queries, locks, IOPS.

Without this you see the symptom (“got slow at 800 RPS”) but not the cause (“the pool of 20 connections is exhausted”).

Environment

A test on a stand 10× weaker than production gives numbers that mean nothing. You need a prod-like configuration (resources, data, topology, network) — or at least a clear scaling factor. Otherwise you’re testing the stand, not the system.

Coordinated omission — the silent killer of percentiles

A classic measurement error (described by Gil Tene): if the load generator waits for a response before sending the next request, then during a stall it doesn’t send the requests it should have — and doesn’t count their large delays. As a result p99 looks great while reality is awful. Fixed by the open model / a coordinated-omission correction in the tool.

Tools: when to use which

  • k6 — scriptable in JS, native open model, great for CI and as code.
  • Gatling — JVM/Scala(/Java), high throughput from one machine, good reports.
  • Locust — Python, scenarios as code, convenient for Pythonistas and distributed runs.
  • JMeter — mature, GUI, many protocols; harder to version scenarios.

Common mistakes

  • Average instead of percentiles.
  • A test on an empty DB / warmed data unlike production.
  • Closed model only → problem masked.
  • One endpoint instead of a traffic profile.
  • No server-side metrics → can’t find the bottleneck.
  • Ignoring error rate (green with 5xx).
  • No warm-up and ramp-down.
  • Coordinated omission not accounted for.
  • A single run without repeatability — noise taken as a result.
  • The stand isn’t prod-like.

Load test checklist (12 points)

  1. The goal and test type are defined (load/stress/soak/spike/breakpoint).
  2. Target SLAs/SLOs are set (p95/p99 latency, max error rate).
  3. Metrics — percentiles + throughput + error rate + saturation, not averages.
  4. Open model where you measure throughput.
  5. Realistic traffic profile (mix of endpoints and proportions).
  6. Ramp-up + warm-up; measurement on the stable section; ramp-down.
  7. Prod-like data (volume, indexes) and environment.
  8. Server-side metrics captured (CPU/mem/IO, DB pool, queues, GC).
  9. Coordinated omission accounted for.
  10. Error rate is part of the pass criteria, not just latency.
  11. The test is repeatable (several runs, stable results).
  12. The bottleneck is found and described, not just the symptom.

Bottom line

A load test is valuable not for the number “we held 1000”, but for showing the real ceiling, the tail behavior and the point of failure. Percentiles instead of averages, the open model, prod-like data and server-side metrics — that’s what turns “we poked at the load” into conclusions you can trust before a release.

Sources