Site reliability engineering · Production operations
SRE incident response and remediation how-tos
Run production incidents through recovery, root cause analysis, tested corrective work, and a safe return to steady-state monitoring. These nine operational how-tos explain the technical choices and the enterprise constraints that often determine whether a recovery plan is executable.
Operate a lifecycle, not a single closure checkbox
Restoring a user journey ends the urgent recovery phase when its acceptance criteria are met. It does not prove that recurrence is prevented. Keep the incident, problem investigation, corrective changes, and residual-risk decisions linked but independently trackable. Investigation, early warning, and implementation often overlap; this is not a waterfall that postpones detection until an RCA is complete.
- Incident: establish user harm, temporary command, authorized operators, and a shared action record.
- Remediation: contain the failure, restore correct service, reconcile accepted work, and state remaining restrictions.
- RCA: reconstruct the mechanism and contributing conditions, including missing responders and access delays. Preserve uncertainty where evidence is incomplete.
- Instrumentation and early warning: test user-path, mechanism, telemetry-health, and notification signals with the receiving team.
- Fixes in progress: retain bounded temporary controls while design, security review, vendor work, and deployment are pending.
- Tracking and verification: move through proposed, approved, implementing, deployed, and verified states with evidence and an accountable owner.
- Retiring temporary alerting: remove redundant incident watches only after replacement coverage or prevention passes and the receiving operator accepts the handoff. Keep normal service monitoring.
All scenarios in this series are hypothetical enterprise situations. They explain actions, constraints, and acceptance tests rather than claim actual customer outcomes.
Nine enterprise SRE how-to articles
How to run enterprise incident command
Create a working command structure before parallel changes make the outage harder to understand. Keep mitigation moving when the service owner cannot be reached or the people on the bridge cannot access production.
How to replace noisy alerts with actionable SLO monitoring
Improve detection without simply moving an uncomfortable signal out of the pager. Tie each page to a user outcome, an operating action, and the team that can take it.
How to manage releases after an error budget is exhausted
Turn an exhausted error budget into a bounded release decision. Separate the business value of a launch from its failure exposure, and make exceptions traceable through the product ownership chain.
How to contain a failing downstream API
Protect a production service when a downstream API is slow, inconsistent, or returning incomplete results. Control how much work crosses the dependency boundary and define what happens to work already accepted.
How to recover a deployment that fails after rollout
Recover from a release whose canary passed but whose full rollout fails later. Compare the changed path, choose a recovery action compatible with the data, and test the accumulation mechanism that the canary missed.
How to detect production failures your monitoring missed
Find the observation gap when customers report failures but the dashboards remain green. Reproduce the affected path, bound the impact honestly, and test a detector that does not depend on the same failed assumption.
How to prevent recurring incidents after the RCA
Use repeated incidents to find the control that never changed. Keep the analysis grounded in evidence, make preventive work verifiable, and stop treating a closed incident ticket as proof that recurrence is addressed.
How to build a two-quarter enterprise reliability program
Make reliability work survive roadmap pressure, team boundaries, and leadership changes. Start with the customer journeys and shared risks that justify investment, then connect every priority to capacity, an owner, and evidence of reduced exposure.
Take decisions through the service and product ownership chain
Use existing delegated authority and name the people carrying each decision. Temporary incident command coordinates the response; it does not replace security approval, change authorization, or business risk ownership. When the primary person is unreachable, escalate to the published backup and duty authority. If no delegate can authorize an action, record that constraint and use feasible approved containment.
| Role | Accountability | Escalation or handoff |
|---|---|---|
| Incident commander | Coordinates workstreams, response priorities, updates, and handoffs. | Duty engineering authority appoints a temporary lead; receiving leads explicitly acknowledge handoff. |
| Service owner / engineering manager | Explains system constraints and owns recovery and technical corrective work. | Escalate missing operators, blocked dependencies, and overdue verification to engineering leadership. |
| Product owner | Defines acceptable customer behavior and prioritizes corrective work against roadmap commitments. | Escalate conflicting product goals and missing improvement capacity to portfolio leadership. |
| Portfolio / authorized business owner | Arbitrates cross-product capacity and accepts business trade-offs within their mandate. | Security, contractual, and customer authorities remain involved where their approval is required. |
| Security and change authorities | Approve scoped access, evidence handling, exceptions, and privileged changes under policy. | Use documented emergency delegation with expiry and audit; an urgent bridge request is not an access grant. |
| Problem owner and receiving operator | Keep implementation, testing, monitoring, and operating acceptance connected. | A reorganization or product-owner change requires acknowledged transfer of open problems, exceptions, and alert ownership. |
Keep an explicit decision record: feasible options, affected journeys, cost of waiting, approving authority, expiry or review date, operator, and recovery evidence. Silence is not approval. A blocked action should remain visible with its dependency and next escalation, rather than disappear into a generic team queue.
Track fixes beyond deployment
A problem owner keeps the incident’s prevention work connected to the product backlog, authorized change, deployed configuration, and verification. Product ownership changes must transfer those commitments along with feature work. If the receiving owner defers the fix, record residual exposure, the temporary protection, and the next risk decision.
Problem: [mechanism, affected journey, linked incident IDs]
Accountable owner and backup: [people / operating roles]
Product sponsor and risk authority: [decision owners]
State: proposed / approved / implementing / deployed / verified
Blocked dependency: [access, vendor, capacity, approver]
Next milestone: [date, acceptance evidence, escalation if missed]
Temporary control or alert: [purpose, operator, review date]
Verification: [failure test, fleet scope, recovery and data checks]
Retirement: [replacement coverage, operating acceptance, rule revision]Do not mark an action verified because a pull request merged. Prove deployment to the affected cohort and test the original failure within approved conditions. Keep the resulting artifacts within the required retention and security boundary.
Apply the how-tos to production mechanisms
- Kubernetes disk pressure: use recurring-incident prevention to bound collector storage and verify behavior during a logging-sink outage. Disruption budgets do not prevent kubelet node-pressure evictions.
- Delayed cloud egress failure: use deployment recovery to distinguish connection churn and shared SNAT exhaustion from an application-version correlation.
- Hybrid-path failure: use cross-team coordination to test real payloads across ExpressRoute and VPN paths when routing sessions and pod checks stay healthy.
- Alert arithmetic: use multiwindow SLO alerting to distinguish noisy symptoms from customer-impact burn and missing measurements.
Verify the path when people and permissions are missing
Schedule bounded exercises that include an unavailable primary responder, a denied broad-access request, a vendor delay, a product owner who cannot accept the trade-off, and an incomplete handoff. Agree the test authority, blast radius, stop condition, and restoration path before the exercise.
Measure the actual observation, authorization, execution, and acknowledgment delays. A healthy technical recovery command is insufficient when nobody on shift can execute it. Feed the results into access design, response delegation, and product capacity planning.
Documentation and operational references
Each article links its relevant primary references. Start with Google’s incident-response guidance, SLO alerting, and the example error-budget policy. Platform-specific behavior must be checked against the actual version, topology, and approved operating policy.