How to detect production failures your monitoring missed

1. Reconstruct the customer journey and bound the impact

Obtain the affected origin, identity class, tenant, operation, time range, and a safe correlation identifier. Reproduce the actual path with authorization; a successful request from the responder’s laptop may use a different DNS answer, proxy, identity, or region.

Reconcile edge arrivals, application attempts, completions, and durable business outcomes. A missing completion can reflect failure, an in-progress workflow, or delayed telemetry. Deduplicate by operation ID and account for asynchronous completion before counting loss. Preserve the first known failure and the interval in which onset remains uncertain; do not replace missing evidence with precise-looking outage statistics.

Illustrative scenario: a private enterprise upload works inside Azure but stalls for on-premises users after VPN failover. Existing probes originate inside the cloud and use tiny unauthenticated payloads. They establish only that their own path works. Compare controlled payload sizes and both origins while the network team tests the failover path.

2. Identify which shared assumption made detection fail

Draw the detector’s own dependencies: execution origin, DNS, credentials, network route, telemetry collector, storage, rule evaluation, paging provider, and receiving team. A synthetic using the same cluster and collector as the application can disappear with it. An external probe may provide independence but fail to represent a private path.

Choose complementary observation points. Use a representative business-path check where it is allowed, application correctness metrics for diagnosis, and an independently delivered measurement-pipeline heartbeat. “No data” needs its own operating meaning. Querying an empty log index is not evidence that customers experienced no failures.

Define success semantically. Authentication, mandatory result fields, acceptable latency, and durable completion may all matter. A successful landing page does not validate an enterprise order journey. Avoid one synthetic that traverses every dependency if it cannot identify which team can act; pair the journey detector with focused diagnostics.

3. Design useful probes within security and privacy limits

If InfoSec prohibits synthetic writes in production, propose a read-only authenticated path, an approved test tenant, or a transaction with an established reversible lifecycle. Document which side effects are excluded. A read-only probe cannot prove payment settlement; preserve the business reconciliation that covers the missing step.

Give synthetic identities least privilege, credential rotation, quotas, and a named service owner. Place evidence in the approved boundary and redact customer payloads. Limit metric dimensions; transaction IDs belong in controlled logs or traces rather than unbounded metric labels.

If the identity or network owner is unreachable, deploy only detection available within existing authority and record the coverage gap as residual risk. Request the minimum specific access and escalate through the duty manager and security delegate. Do not create an undocumented shared account to make the new check pass.

4. Agree the detection contract with product and operations

The product owner identifies the customer harm and latest useful intervention time. The service owner specifies the responding team, runbook, and dependencies. For low-volume but high-value customers, a traffic-ratio alert may be insufficient; a missed scheduled outcome or a dedicated authorized probe can be more meaningful.

When the budget for a remote probe or private network collector crosses product boundaries, escalate a concrete proposal: coverage gained, uncovered paths, operating cost, and owner. If funding is deferred, preserve a bounded manual reconciliation or incident watch with a review date. Do not quietly redefine the SLO to exclude the customer cohort that is expensive to observe.

Support and customer-success teams need an approved message that distinguishes detected symptoms, estimated impact, and unresolved transactions. A newly functioning alert does not reconcile a customer’s missing operation.

5. Inject the original failure and verify notification delivery

In an approved representative environment or bounded production exercise, reproduce the missed mechanism: drop the route, remove required response data, delay the collector, or expire the test credential. Confirm the intended alert fires, reaches the right operator, gives a useful action, and clears after actual recovery. Each injection needs its own stop and recovery conditions.

Measure detection from known injected impact and distinguish it from notification and acknowledgment delays. Include no-traffic and maintenance behavior. Test failback as well as failure; a probe may remain pinned to a backup path after the application returns to normal.

Keep the incident watch until the new coverage is deployed, exercised, and acknowledged by the operating owner. Record source revisions and evidence in the problem record. If verification slips, escalate the remaining blind spot; a successful detector deployment is not the same as a passed failure test.

Carry the work through the incident lifecycle

Recovery, investigation, instrumentation, and implementation can overlap. Keep service recovery and the underlying problem’s verification as separate states; reopen incident coordination if customer harm returns.

Technical references

Use the documentation for your deployed version and actual topology. Scenarios are illustrative; proposed checks and thresholds require an approved exercise plan.

• Google SRE Workbook: alerting on SLOs
• Azure ExpressRoute and VPN failover architecture
• Prometheus alerting rules