How to manage releases after an error budget is exhausted

1. Validate the budget and the existing policy

Confirm the SLO period, eligible population, bad-event counts, and data completeness. A quarterly budget cannot be reset by switching a dashboard to seven days. If the measurement is broken, report uncertainty and repair it; do not treat missing data as a reason to resume all changes.

Read the agreed policy with the service owner. A pause on risk-increasing changes is not a ban on security patches, recovery work, or all unrelated services. Identify the affected journey and shared dependencies. A launch in another product can still increase load on the same authentication or egress boundary.

2. Separate release risk by failure mechanism

Review code, configuration, infrastructure, migrations, and traffic shifts together. A small application diff can create a large connection multiplier. A reversible feature flag may contain behavior, but only if disabling it stops new work and leaves already accepted work recoverable.

Classify alternatives: defer the launch, remove the risky dependency, release to an isolated cohort, or first ship the corrective control. Compare expected user harm, detection time, rollback compatibility, data integrity, and responder availability. A low-traffic canary does not exercise a shared SNAT capacity threshold merely because its pods remain healthy.

Illustrative scenario: a product wants a bulk-import launch after repeated outages. Imports share the production database and would create non-reversible side effects. A read-only preview may be a separate low-risk launch; enabling writes requires a bounded queue, capacity proof, idempotency, and a reconciliation plan.

3. Escalate an explicit choice through product ownership

The service owner supplies feasible recovery and containment options. The product owner states the launch’s business deadline and acceptable degraded behavior. When shared infrastructure affects several product owners, the engineering and portfolio leaders arbitrate capacity and risk at that shared boundary.

Record the requested exception, alternatives rejected, affected customers, approving authority, expiry, operational coverage, and conditions that cancel approval. Product prioritization does not authorize access-control changes or waive contractual commitments. Include InfoSec, compliance, legal, or the customer authority when their designated approval is required.

If the decision-maker is unreachable, use an established delegate or remain within the existing release policy. A request marked urgent does not create a delegate. Escalate the consequence of waiting with facts, including deadlines, rather than inventing agreement from silence.

4. Make the release gate operationally enforceable

Before the change, confirm an authorized operator, available backup, known-good artifact and configuration, and a tested stop or recovery path. If security policy prevents direct access, use the approved pipeline and an authorized platform operator. An exception should not depend on credentials that nobody on shift can obtain.

Define cohort, concurrency limit, minimum observations, error and queue-age boundaries, and a maximum observation time. Include the data-plane action required to stop the experiment; pausing a Deployment does not undo accepted writes. If rollback could corrupt data, the plan may require stopping intake and reconciling before roll-forward.

Release exception: EXAMPLE-123
Affected journey and SLO: [scope, period, observed budget]
Approved change and cohort: [artifact, config, migration boundary]
Business decision owner: [named authorized approver or delegate]
Execution and backup owners: [authorized operators]
Stop / recovery conditions: [measurable limits and actions]
Expiry and review: [time, decision route if the fix slips]
Evidence: [change, test, incident, problem, and risk records]

5. Review the outcome without erasing the debt

Verify the critical journey and any backlog or data reconciliation after release. Successful promotion does not restore a consumed budget. Report the exception’s outcome to the approving owner, preserve failed hypotheses, and update the original problem record.

Keep reliability work visible in the product backlog with reserved capacity and acceptance criteria. If a roadmap change defers the fix, require an explicit residual-risk review and a new date. Repeatedly extending a temporary watch is evidence that the organizational control is failing, even when the page itself works.

Carry the work through the incident lifecycle

Recovery, investigation, instrumentation, and implementation can overlap. Keep service recovery and the underlying problem’s verification as separate states; reopen incident coordination if customer harm returns.

Technical references

Use the documentation for your deployed version and actual topology. Scenarios are illustrative; proposed checks and thresholds require an approved exercise plan.

• Google SRE Workbook: example error-budget policy
• Kubernetes Deployments and rollback behavior