How to build a two-quarter enterprise reliability program

1. Establish a journey and dependency baseline

For an illustrative estate of 40 services across 12 teams, do not begin by assigning the same SLO to every service. Identify the journeys whose failure has the greatest customer, financial, contractual, or mission impact. Map identity, data, networking, CI/CD, and observability dependencies shared across them.

Collect incident timelines, budget consumption, support escalations, change failures, on-call load, toil hours, and unverified corrective actions. Preserve low sample counts and severity. A lower median recovery time can coexist with a catastrophic tail; changing incident definitions can also create a false improvement.

Name a service owner, product owner, authorized change path, and backup for each selected journey. Include support hours and vendor dependencies. An inventory filled with team names is incomplete when nobody can accept an escalated problem or authorize the recovery action.

2. Make the ownership and capacity agreement explicit

The service owner defines technical risk and recovery constraints. Product owners prioritize customer outcomes and delivery commitments. Engineering leadership provides implementation and on-call capacity. Portfolio leadership arbitrates shared investments when one team pays for a control that benefits several products. Security participates in access, evidence, and control designs that require its authority.

Fund a small set of workstreams against the highest shared risks rather than a platform-wide dashboard project. For each, record the failure mechanism, affected journeys, proposed control, acceptance exercise, cost, accountable owner, dependencies, and milestone. Reserve an explicit amount of improvement capacity through planning; the percentage is a negotiated allocation, not a universal SRE rule.

If a sponsor or product owner changes, require a receiving owner to acknowledge open reliability commitments and risk exceptions. If no receiving owner accepts, escalate to the portfolio owner. Keep the work visible as blocked; do not close it because the original team was reorganized.

3. Sequence two quarters around verifiable outcomes

Use the first quarter to make the most important risks measurable and recoverable. Use the second to remove the repeat mechanisms and prove the controls under failure. Adjust scope when the baseline reveals a larger problem rather than pretending the original dates are unaffected.

A hybrid exercise must include actual origin, identity, representative payload, primary/backup routing, and failback. Healthy route sessions alone do not satisfy it. A deployment exercise needs compatible configuration and data recovery, not merely a command that restores an image.

4. Plan for security restrictions and incident interruptions

Bring access and evidence requirements to InfoSec during design: scoped diagnostic roles, approved synthetic identities, collection boundaries, and emergency-change delegation. A program that discovers its required tests are prohibited in the final week has not planned the work realistically.

When an access request is denied, track the reason, acceptable alternatives, decision owner, and resulting verification gap. Use representative lower environments or authorized operators where useful, but state what those substitutes cannot prove. If a vendor is the only team that can perform a required action, incorporate its support hours, contract, and escalation time into the recovery design.

Incidents will interrupt delivery. Make the capacity trade-off visible: which corrective action moved, what temporary protection remains, and which product or portfolio owner accepted the delay. Keep urgent security work and actual customer recovery distinct from discretionary launch pressure; do not use an error-budget policy as a blanket excuse to stop unrelated work.

5. Measure exposure reduction and stop low-value work

Review budget consumed by repeat mechanisms, actionable pages per shift, toil hours, change failure rate with release volume, and recovery results by severity. Track acknowledgment and access delays separately from technical execution so the right intervention is funded. Avoid a scorecard that rewards suppressing incidents, narrowing eligible events, or reducing release count without considering delivery value.

Stop duplicate status reporting, pages with no useful response, and manual repair that a tested control can replace. Keep diagnostics that answer an operating question even if they do not page. Reallocate time to prevention and exercises with the affected teams’ agreement.

At each portfolio review, distinguish proposed, approved, implementing, deployed, and verified work. Report blocked dependencies and expiring exceptions alongside completed milestones. A fixed-count target such as “close 20 RCA actions” is weaker than proving that the most harmful failure mechanisms are now bounded.

6. Build alert retirement and ownership review into delivery

Every temporary watch needs an owner, an operating action, a review date, and an exit test tied to a control. Budget its continuing response cost while the fix is delayed. Review dates cause escalation or renewal; they must not automatically remove the last detection path.

Retire the watch after the prevention or permanent detection passes and the receiving team accepts operations. Retain steady-state SLO coverage, required audit retention, and evidence of the retirement decision. Revisit this coverage when services, product ownership, routes, or dependencies change; yesterday’s redundant alert may have covered a path that was silently reintroduced.

Carry the work through the incident lifecycle

Recovery, investigation, instrumentation, and implementation can overlap. Keep service recovery and the underlying problem’s verification as separate states; reopen incident coordination if customer harm returns.

Technical references

Use the documentation for your deployed version and actual topology. Scenarios are illustrative; proposed checks and thresholds require an approved exercise plan.

• Google SRE Workbook: SLO implementation
• Google SRE Workbook: example error-budget policy
• Google SRE Workbook: incident response