Site reliability engineering · Production operations
How to prevent recurring incidents after the RCA
Use repeated incidents to find the control that never changed. Keep the analysis grounded in evidence, make preventive work verifiable, and stop treating a closed incident ticket as proof that recurrence is addressed.
1. Compare the mechanism across incidents
Build a timeline for each recurrence using user impact, configuration changes, workload, dependency behavior, operator actions, and recovery evidence. Separate observations from hypotheses and mark missing telemetry. Similar symptoms can have different causes; prove the common mechanism before grouping every outage into one RCA.
Review the previous corrective actions as deployed behavior. Was the code merged, approved, rolled out to every relevant cluster, tested under failure, and accepted by the receiving team? “Implemented” may mean a pull request exists while the affected pool still runs the old configuration. Preserve these state distinctions in the problem record.
Use blameless analysis to understand how reasonable local choices interacted. An unavailable owner, denied diagnostic access, an unreliable runbook, and a load assumption can contribute without being a single technical root cause. Do not claim certainty that the retained evidence cannot support.
2. Replace an unbounded failure path with a real boundary
Illustrative scenario: a collector destination becomes unavailable, buffers grow, and node filesystem pressure evicts critical pods. The prior two fixes increased disk size and added a cleanup runbook. Both delayed exhaustion without bounding file growth or defining what happens when the sink stays down.
Inspect the actual storage layout and the process producing files. Container writable layers, disk-backed emptyDir, and a collector’s host-mounted buffer can have different accounting. Distinguish bytes from inodes and distinguish eviction from OOMKilled. Kubernetes PodDisruptionBudgets do not prevent kubelet node-pressure eviction.
Bound collector buffering, application log production, and retention at their real control points. Configure supported application ephemeral-storage requests and limits, log rotation, and an explicit overflow behavior. A pod storage limit does not automatically constrain an unrelated host-mounted buffer; storage limits can also lead to eviction rather than reserve usable disk space.
Choose overflow behavior with the data owner. Dropping diagnostic logs may be acceptable under a defined sampling policy; losing required audit records may not be. Involve security and compliance in the buffer, backpressure, archive, or intake decision. Never make room by deleting protected evidence during the incident.
3. Define a prevention test before closing the action
Recreate the sink outage on a representative test pool for longer than the original accumulation interval, within an approved exercise plan. Include inode pressure, sustained traffic, collector restart, backlog replay, and return of the destination. Verify the critical journey and the required logging behavior together.
Set measurable criteria: buffer consumption remains bounded, the agreed audit-loss or delivery contract holds, no uncontrolled drain overloads the destination, and warning arrives with enough time for the authorized intervention. A new dashboard or a disk expansion is not sufficient acceptance evidence for an unbounded producer.
Corrective action: [problem ID and specific unsafe condition]
Accountable owner: [service owner, not an unowned team queue]
Dependencies: [platform, security, vendor, release approval]
State: proposed / approved / implementing / deployed / verified
Acceptance test: [sink outage, duration, workload, storage boundary]
Evidence: [revision, cohort, results, reviewer acknowledgment]
Residual risk: [temporary guardrail, review date, escalation owner]4. Escalate blocked corrective work through the product backlog
The engineering owner estimates implementation and verification work, including platform or vendor dependencies. The product owner reserves capacity and understands the customer harm from another recurrence. A date without capacity or a receiving team is not a delivery commitment.
If the product owner leaves, the service changes business unit, or the roadmap drops the fix, require an acknowledged transfer of the open problem and risk. Escalate overdue high-impact work to the engineering and portfolio owners with incident recurrence, current exposure, and feasible alternatives. Do not silently reset the due date or remove the risk from the next report.
If InfoSec denies the proposed control because it changes audit retention or access, request a design review with alternatives and acceptance evidence. Keep the approved temporary buffer cap, capacity watch, or controlled intake limit while that decision is pending. Missing approval is a tracked dependency, not a reason to claim the unsafe condition has been fixed.
5. Close verified work and retire only redundant detection
Roll the change out gradually and verify the relevant fleet inventory. Confirm actual configuration as well as deployment success. The problem owner reviews the failure-test evidence, required operating changes, and residual risks before marking the preventive action verified.
Remove incident-specific manual disk checks and duplicate paging after the permanent bounded behavior and warning signal pass. Keep the user-journey SLO and the storage signal still needed to operate the system. Store the retired rule revision and reason so a future responder understands why it was removed.
A quiet week alone is weak proof when the sink failure occurs monthly. Require the controlled failure test and an observation window covering relevant operating cycles. If the warning remains the only protection, its removal is a new risk decision rather than routine cleanup.
Carry the work through the incident lifecycle
Recovery, investigation, instrumentation, and implementation can overlap. Keep service recovery and the underlying problem’s verification as separate states; reopen incident coordination if customer harm returns.
| Stage | Required work and closure evidence |
|---|---|
| Incident | Link the recurrence to prior incidents while testing whether the mechanism is genuinely the same. |
| Remediation | Recover safe capacity and bounded behavior without deleting protected evidence. |
| RCA | Reconstruct the mechanism, prior incomplete actions, access constraints, and organizational contributing conditions. |
| Early warning | Monitor the actual pressure boundary and user impact with actionable lead time. |
| Fixes in progress | Track implementation, security review, rollout, and residual guardrails as distinct states. |
| Tracking and verification | Require deployed fleet evidence, a representative failure test, and an accountable reviewer before verified closure. |
| Retiring temporary alerts | Remove duplicate incident watches after prevention and replacement warning pass; retain required steady-state signals and evidence history. |
Technical references
Use the documentation for your deployed version and actual topology. Scenarios are illustrative; proposed checks and thresholds require an approved exercise plan.