How to run enterprise incident command

1. Establish impact, authority, and a change record

Open an incident record with the affected user journey, first confirmed impact, scope, severity, and the next update time. Distinguish observed failures from suspected causes. “Orders accepted but not confirmed” is a more useful operating statement than “Kubernetes is unhealthy.” Attach representative transaction IDs in the approved evidence system, not a public chat.

Assign an incident commander, a technical lead, an authorized change operator, a communications lead, and a scribe. A small incident can combine roles; a large one needs explicit handoffs. The commander coordinates priorities and escalation. That appointment does not grant production access, permission to bypass security controls, or authority to accept every business risk.

Use a shared action log: hypothesis, proposed action, target, operator, start time, expected effect, stop condition, and observed result. Require coordination for changes that affect the same failure domain. Parallel read-only investigations are useful; independent routing, application, and database changes can erase the comparison needed to judge recovery.

2. Continue safely when the right people are unreachable

Page the published secondary and then the duty manager through the existing escalation policy. Set a bounded acknowledgment interval appropriate to the incident; record failed contact attempts. If nobody accepts the role, escalate to the engineering duty executive for a temporary coordinator and an authorized operator. Do not let the bridge infer that the person who speaks most confidently owns the service.

While waiting, collect evidence available within existing permissions and select preapproved containment: pausing promotion, suppressing an optional feature, or limiting intake where the relevant operator is authorized. Verify the runbook version and current topology. A stale runbook can direct traffic into a retired dependency.

For a shift handoff, the receiving lead explicitly acknowledges current impact, active changes, outstanding access requests, the next decision deadline, and who is still being paged. Posting a transcript without an acknowledgment leaves command ambiguous.

3. Request the smallest useful diagnostic capability

If InfoSec denies unrestricted cluster access, ask for the specific observation needed: read-only events for one namespace, a redacted trace, an approved saved query, or an authorized platform operator running a documented command. State the diagnostic question and retention boundary. Broad access is often slower to approve and exposes more data than the investigation needs.

Use the established emergency-access process when justified, including its approver, scope, expiry, audit trail, and revocation. If the approver is unavailable, use the policy’s delegated path; do not share credentials or treat incident severity as a permission grant. Record an unresolved denial as an operating constraint, explain the impact on recovery time, and pursue a lower-access mitigation.

Illustrative scenario: the cloud team can inspect an AKS node pool, but the application team cannot. A recent logging change correlates with evictions. The platform operator exports selected eviction events while the application operator reverts its compatible logging configuration. One change is measured at a time. Removing all node logs would destroy evidence and might violate retention obligations.

4. Take degraded-service choices up the ownership chain

Ask the service owner which technical recovery options are feasible. Ask the product owner which user journeys can degrade: disabling recommendations may be acceptable; accepting orders without a durable record may not be. Present options with affected cohorts, estimated recovery time, data-integrity implications, and a stop condition.

When product owners disagree, escalate the explicit choice to the portfolio or business owner authorized to arbitrate it. Security or privacy exceptions need their designated authority as well. If the decision-maker is absent, stay within delegated authority and make the limitation visible. “Product wants it” is not a recorded risk decision.

Publish impact, the action underway, confidence, and the next update even when no new cause is known. Avoid publishing an unverified restoration time as a commitment. Reserve investigators’ time by routing stakeholder requests through the communications lead.

5. Declare recovery with evidence and preserve unfinished work

Check completion and correctness from the affected customer paths over an agreed observation period, including backlog age, replay or reconciliation, and remaining regional or tenant impact. A recovered pod count is supporting evidence, not the service’s exit criterion.

Close active command when recovery is accepted and residual restrictions have owners. Keep separate problem, change, and risk records open. Capture the response delay caused by unreachable people or denied access as a condition to improve, with the actual policy and timing; do not reduce the RCA to blaming the absent responder.

Carry the work through the incident lifecycle

Recovery, investigation, instrumentation, and implementation can overlap. Keep service recovery and the underlying problem’s verification as separate states; reopen incident coordination if customer harm returns.

Technical references

Use the documentation for your deployed version and actual topology. Scenarios are illustrative; proposed checks and thresholds require an approved exercise plan.

• Google SRE Workbook: incident response
• Kubernetes node-pressure eviction