How to recover a deployment that fails after rollout

1. Stop promotion and capture the changed system

Pause further promotion through the authorized deployment system and open incident command when impact warrants it. Record image digests, configuration revisions, feature flags, migrations, routing changes, and the time each cohort received them. A Deployment revision does not capture every external ConfigMap, secret, database, or upstream configuration change.

Capture representative failed and successful traces and user outcomes. Compare versions within equivalent node, zone, tenant, and request cohorts. A comparison of new pods in a busy pool against old pods in a quiet region confounds version with load and placement. Time-box diagnosis while planning a safe recovery in parallel.

# Read-only examples: use the approved context and namespace.
kubectl -n orders rollout history deployment/checkout
kubectl -n orders get deployment checkout -o yaml
kubectl -n orders get pods -l app=checkout -o wide
kubectl -n orders get events --sort-by=.metadata.creationTimestamp

Deployment YAML can contain operationally sensitive values; use the approved evidence store and redact before sharing. These commands observe rollout state. They do not authorize a production change.

2. Test a mechanism that needs time or shared load

Investigate connection churn, file growth, cache expiry, credentials refreshed on a timer, scheduled batches, and downstream quotas. A canary can be healthy because it has not existed long enough, has different requests, or never reaches a shared threshold. Its percentage of pods is not proof that it represented those conditions.

Illustrative scenario: an AKS application uses Standard Load Balancer outbound rules. A new client stops reusing partner-API connections. The canary passes at low load; full rollout plus retries produces delayed SNAT exhaustion. Confirm the actual outbound route before selecting metrics: NAT Gateway, a proxy, a firewall, and Standard Load Balancer have different allocation behavior.

For that specified Load Balancer path, inspect failed SNAT connections and Used versus Allocated SNAT Ports by backend IP and protocol. Correlate with connection creation, retry attempts, and partner completion. Fleet averages can hide one exhausted backend. Scaling replicas may increase egress demand rather than relieve it.

3. Select rollback, isolation, or roll-forward

If the changed client configuration plausibly explains continuing harm and the previous configuration is compatible, restore it through the approved release path. Do not wait for complete causal certainty while avoidable harm continues. At 8% eligible failures against a 99.9% success SLO, burn is 80×; convert this into budget consumption using actual traffic and the agreed period.

Rollback is unsafe when the previous application cannot read the new schema, accepted writes cannot be reversed, or side effects have already escaped the system. Isolate the feature or intake, preserve operation IDs, and use the migration/reconciliation plan. Confirm what a feature flag stops: it may prevent new requests without cancelling active workers.

Kubernetes progressDeadlineSeconds reports stalled Deployment progress; it does not itself perform automatic rollback. Pausing a rollout is also not reverting it. Restoring an old image without its compatible configuration can preserve the defect or create a second incident.

4. Resolve unavailable approvers and denied production access

If the pipeline owner is unreachable or credentials were revoked, escalate to the authorized platform backup and emergency-change authority. Ask InfoSec for the smallest approved action: execution of a known signed artifact or a scoped pipeline role, not a shared administrator token. Record the request, decision time, and what can be mitigated under existing permissions.

The product owner decides whether a degraded feature or paused intake is acceptable within their mandate. Engineering explains data and recovery constraints; product cannot make incompatible data reversible by accepting business urgency. If several products share the dependency, escalate the trade-off to the portfolio owner rather than letting the loudest release sponsor control the bridge.

Do not close the incident because rollback execution succeeded. An authorized operator may report a completed deployment while stale connections, queued work, or a subset of clients still fail. Communications should distinguish “change applied” from “customer recovery verified.”

5. Turn the missed canary condition into an acceptance test

Observe the affected journey over a period covering the measured accumulation or refresh cycle. Check that failed outbound flows and retry amplification recover, backlog age falls within the contract, and no new cohort regresses. Keep a documented trigger to stop the recovery experiment itself.

Assign a problem owner to client pooling, retry limits, and representative shared-capacity testing. Test the fix under realistic connection lifecycle and failure behavior before gradual rollout. While approval or platform changes are pending, retain the temporary concurrency cap and an owned watch with a review date. Do not repeatedly extend the exception without a product-level prioritization decision.

Carry the work through the incident lifecycle

Recovery, investigation, instrumentation, and implementation can overlap. Keep service recovery and the underlying problem’s verification as separate states; reopen incident coordination if customer harm returns.

Technical references

Use the documentation for your deployed version and actual topology. Scenarios are illustrative; proposed checks and thresholds require an approved exercise plan.

• Kubernetes Deployment progress and rollback
• Azure Load Balancer diagnostics
• Azure outbound connection troubleshooting