non-functionalobservabilitymonitoringsloqa

Observability for QA: logs, metrics, traces — what to test and how to use it

“The bug only reproduces in production, once a day, and only for some users.” Without observability, a bug like that means hours of guesswork. With it — a few minutes: open the trace of the failed request, see which service failed and why. Observability isn’t a “DevOps thing” — it’s a working tool for QA. Let’s go through the three pillars, how to test observability itself, and how to use it day to day.

Three pillars: logs, metrics, traces

The canonical breakdown is from Peter Bourgon and the OpenTelemetry docs. Briefly, in QA terms:

  • Logs — discrete events (“what happened at this moment”): an error, entering a function, an external call failing. Great for the details of a specific incident. Should be structured (JSON), not a “wall of text”.
  • Metrics — numeric aggregates over time (“how much and how fast”): RPS, error rate, latency percentiles, queues, resource usage. Cheap, great for trends and alerts.
  • Traces — the path of a single request through all services (“where exactly time was spent and where it failed”). Indispensable in distributed systems: they show the whole A→B→C chain with the timing of each step.

The rule: a metric tells you what is wrong, a trace — where, a log — why.

Test observability itself

Observability is a feature, and it gets tested too. What to check:

  • On an error, a log of the right level is actually written with context (what operation, the entity id, the reason) — not “silently swallowed”.
  • There’s an end-to-end correlation/trace id: a single request can be followed across all services by one identifier.
  • Key business and tech events produce metrics (success/error/latency), and they go up on the dashboard when you run the scenario.
  • ⚠️ No secrets or PII leak into logs/traces: passwords, tokens, card numbers, personal data are masked. This is a frequent and expensive bug — check it separately.
  • Log levels make sense: ERROR — needs a reaction; WARN — suspicious; INFO — business events; DEBUG — details. Production doesn’t spew gigabytes of DEBUG.
  • An alert actually fires for a simulated problem (raise the error rate in a test — did the alert arrive?).

How QA uses observability day to day

  • Localize a distributed bug: the trace id shows which of the 5 services the request failed or hung on — no guessing, no pinging every developer.
  • Reproduce from logs: structured logs with context give the exact inputs and sequence that triggered the bug.
  • Catch silent degradations: functionally everything “works”, but p99 latency doubled or the error rate is creeping up — visible only in metrics, not in manual clicking.
  • Verify after a release: compare dashboards before/after the deploy — did errors/latency rise (part of production acceptance, canary).
  • Back a bug with data: not “feels slow”, but “checkout p95 is 4.2s against a 1s SLO, here’s the graph” — a bug report with a metric isn’t arguable.

SLI / SLO / error budget

From the Google SRE Book. It’s the language a team uses to agree on what “reliable enough” means:

  • SLI — an indicator metric (e.g., the share of requests faster than 300ms, the share of successful responses).
  • SLO — a target for the SLI (e.g., 99.9% of requests < 300ms over 30 days).
  • Error budget — the allowed “budget” of violations (0.1% = how much you can fail). While there’s budget — ship features; budget exhausted — stop and fix reliability.

For QA this means concrete acceptance criteria for non-functional requirements: not “should be fast”, but “p95 < 300ms at the given load”. Testable.

Pitfalls

  • Metric cardinality: a high-cardinality tag (user_id, request_id in metric labels) blows up storage and cost. High cardinality belongs in logs/traces, not in metric labels.
  • Noisy alerts: if it alerts on everything, people stop reacting (alert fatigue). Alert on user-facing symptoms (SLO violations), not on every CPU twitch.
  • Unstructured logs: you can’t filter or aggregate a “wall of text”. Structured only (key-value/JSON).
  • Trace sampling: in production traces are often sampled (e.g., 1%) — a rare bug may not be in the sample; keep that in mind while investigating.

”Observability-ready feature” checklist (for acceptance)

  • Errors are logged at ERROR level with context (operation, id, reason)
  • An end-to-end correlation/trace id passes through all involved services
  • Key events produce metrics: a success/error counter + latency
  • SLI/SLO defined for the critical path + a testable threshold (e.g., p95 < N ms)
  • Secrets and PII don’t reach logs/traces (explicitly checked)
  • Log levels are correct, no DEBUG spam in production
  • There’s an alert for an SLO violation, and it actually fires on a simulated problem
  • Metric labels have no high cardinality (no user_id/request_id in labels)
  • A dashboard shows the feature: you can see its traffic/errors/latency

Bottom line. Observability is a lever for QA: a metric shows what’s wrong, a trace — where, a log — why. Test observability itself (logs, trace id, metrics, no PII in logs) as a full part of the feature, and use it to localize distributed bugs, catch silent degradations, and back findings with data rather than feelings.

Sources: Google SRE Book — Service Level Objectives, OpenTelemetry — Signals (logs/metrics/traces), Peter Bourgon — Metrics, tracing, and logging.