non-functionalsecurityobservabilityqa

Tokens in logs: how QA can test diagnostics for sensitive data leaks

Login works. Other users’ orders are inaccessible. Signing out closes the session. Then a request times out, the HTTP client attaches the request to an exception, and the error handler forwards it to diagnostics. Including the Authorization header.

This is a hypothetical example, not a report of a particular incident. It illustrates a separate testing concern: what the application reveals about its users while trying to explain a failure. Functional authentication tests alone do not answer that question.

The following is a proposed QA test plan. Adapt it to your architecture and agreed data-handling rules. Field names and scenarios are illustrative; you do not need real customer secrets to investigate them.

Agree on what counts as a leak

The OWASP Logging Cheat Sheet advises against directly recording passwords, access tokens, session identifiers, keys and other sensitive values. Personal data needs deliberate handling. Diagnostics must still retain useful evidence, and logging mechanisms themselves need testing.

Turn this into a small contract with the team: required events, allowed fields, forbidden values and the point at which sanitization happens. Recording that authentication failed is useful. Recording the submitted password is not. A separate correlation identifier can link requests without granting access to an account.

Distinguish secrets from identifiers. A bearer token may enable authenticated requests. An internal user ID serves a different purpose, but can still link activity to a person. Decide whether it belongs in each destination. Saying “we hash the email” does not settle what is transmitted, who can see it, or why it is needed.

Write down expectations before testing. Otherwise, QA finds an email in an event, a developer calls it necessary, and the discussion turns into preferences rather than a check against requirements.

Map the journey of one event

Choose a scenario such as session refresh failure. List the destinations involved: application, local log, SDK queue, telemetry receiver, storage, viewer and support export. A server request may also pass through a reverse proxy, application logger, collector, index and alerting system.

Do not assume every component records request bodies. Inspect actual configuration and records. For each destination, identify an owner, an inspection method and a sanitization rule. If you cannot inspect server storage, document that coverage limit. A clean mobile console does not prove every destination is clean.

Separate sanitization before transmission from masking in a viewer. If the viewer shows asterisks but an export returns the original value, that original value is still stored. These are different results and should be reported separately.

Prepare synthetic markers

Use recognizable test values: [email protected] for an email, a unique test password, and another marker in a free-text profile field. Give each data type a distinct marker so a match identifies its origin.

A fabricated access token is useful for an authentication rejection test. It does not exercise successful authentication. For that path, use a session issued to a dedicated test account in a controlled environment. Its token is still a secret: keep it out of public tickets and demonstration screenshots.

Add an independent allowed event marker, such as a correlation_id. This is your positive control: evidence that the exact event reached the destination being inspected. A password missing from an undelivered report tells you nothing about sanitization.

Record the run time, build, environment and SDK configuration. This distinguishes new events from older ones and makes retesting possible. If delivery is delayed, define the observation window using your system’s configuration rather than an arbitrary sleep.

Exercise failure paths separately

Build a small matrix: successful login, incorrect password, expired session, access denial, timeout and HTTP 500. Compare diagnostic fields across the rows. Look for the point where a compact event turns into a complete request dump.

Perform an action successfully, then repeat it with a controlled dependency failure. The application should preserve the reason for failure without suddenly copying credentials because a different error handler ran.

Include response parsing failures. Unexpected text instead of JSON may trigger another handler. In a test environment, supply a response containing a synthetic marker and inspect where it appears. Expectations depend on the contract: some response fields may be allowed, others may require removal.

Trigger a supported test exception for crash reporting. Inspect event fields, extra context and breadcrumbs if your integration collects them. A correctly sanitized application log does not prove that a separate diagnostics channel uses the same filter.

Test data shape as well as field names

Start with a password field, then test credentials.password and arrays of objects where those structures actually occur in your product. Cover supported formats; hundreds of imaginary inputs that never enter the system add little confidence.

For headers, inspect supported representations and name casing. For URLs, examine the path and query separately. For exceptions, check the message and attached context. A secret removed from a structured field may still be repeated inside a free-text string.

A literal string search does not cover every transformation. If the pipeline URL-encodes or serializes values, inspect those representations too. Encoding is not automatically protection: whether the original secret remains recoverable is a separate criterion.

A table of input, destination and expectation helps. For example: synthetic password in a nested object, delivered error event, field absent or replaced by the agreed placeholder. That result is reproducible and does not depend on whether someone likes the appearance of asterisks.

Switch accounts and test delayed delivery

Sign in as A, perform an action, sign out, sign in as B and create a new error. Confirm that B’s new event does not inherit A’s context. Incorrect attribution is a defect even if no password is present.

Repeat with the network disabled. Create A’s event offline, switch accounts, create B’s event and reconnect. An earlier event may legitimately retain the identity captured when it was created. That is not automatically a leak. Check that new B events do not inherit A and that queued events are not accidentally relabelled with the currently active account.

If the SDK persists its queue, include an application restart. Document which events were created before logout and which after it. Without that timeline, legitimate delayed delivery can be mistaken for cross-account contamination.

Check release configuration and support bundles

Repeat selected scenarios with diagnostic settings matching the release. This does not require real customer accounts in production. Use a controlled environment and document differences in logging level, integrations, filtering, sampling and delivery queues.

“Send diagnostics” needs its own test. Generate the bundle, open the archive and inspect its files. Compare them with the viewer: the export may use a separate implementation and contain additional data.

If screenshots are included, inspect visible fields and notifications. Sanitizing JSON does not remove a secret from an image. A synthetic marker is sufficient evidence; real private messages or payment documents are unnecessary.

Inspect your test infrastructure too. Does it copy a detected value into CI output, attachments or assertion messages? A test for secret leakage should not create another copy of the secret.

Automate without false passes

Start with one critical flow: authenticate, trigger a controlled failure, retrieve the event by correlation_id, validate allowed fields and assert that forbidden values are absent. Distinguish three outcomes: clean event, prohibited data found, event not inspected successfully.

The third outcome is not a pass. Sampling, project selection, delivery delay, a disabled SDK or a broken receiver could explain it. Establish delivery first, then inspect content. If you change sampling for deterministic tests, document the difference and also verify release configuration.

Assertion messages should identify the field, event and violation type. They do not need the complete forbidden value. Restrict access to detailed artifacts just as you restrict access to diagnostics themselves.

Expand coverage using actual defects: nested structures, queues, account changes and alternative exception handlers. A generic regex scan is a useful supplementary signal, but cannot replace checks of a specific data route and event contract.

Write an actionable bug report

Include the build, configuration, minimal steps, sensitive field type, destination and expected rule. For example: “After session refresh times out, Authorization is stored under extra.request.headers; the event contract prohibits forwarding this field.”

Use a synthetic marker or a redacted excerpt as evidence. State whether you inspected the outgoing payload, stored event, viewer or export. “Hidden in the viewer” and “never transmitted” are different claims.

Describe impact using the evidence: whether the value grants access, who can read it, how long it remains valid and what copies already exist. Not every identifier is a password; the word debug does not automatically make storage safe either.

After the fix, repeat the original scenario and check diagnostic usefulness. Can you still find the error code, application version, time and related events? Deleting the entire event may satisfy a negative assertion while making the next incident much harder to investigate.

A compact release checklist

  • Allowed and prohibited fields are defined for the critical flow.
  • Synthetic markers and a positive delivery control are available.
  • Success, timeout, authentication rejection and server failure are covered.
  • Nested fields, URLs, exception messages and separate destinations are inspected.
  • Account switching does not contaminate new event context.
  • Delayed delivery and diagnostic exports are covered where applicable.
  • Configuration matches the release, with differences documented.
  • Test reports do not print detected secrets.
  • Sanitized events remain useful for investigation.

Readiness means you can investigate the failure without copying secrets into diagnostics. Both parts need evidence.