Site reliability engineering · Production operations
How to resolve cross-team ownership during an incident
Coordinate a shared failure without waiting for an organizational dispute to be settled. Establish temporary response authority, evidence at the boundary, and a lasting owner for the work that remains after recovery.
3. Work around missing teams without crossing their authority
If the network primary does not acknowledge, use the secondary, duty manager, and contracted provider escalation. Keep the response record explicit about which edge cannot yet be observed. Ask the available team for tests that discriminate without changing that edge, such as size-controlled synthetic transactions or approved flow logs.
If InfoSec will not allow a broad firewall change or packet capture, propose a narrower filter, an approved capture by its operator, or a test environment reproducing the path. Record what uncertainty remains. Do not solve an ownership dispute by issuing a global security exception or disabling inspection controls.
When external support is delayed, give the vendor the shared test matrix and an owner for updates. A vendor case is a dependency in the recovery plan, not a transfer of the incident. Keep customer communications and available containment under internal ownership.
4. Escalate the business decision at the shared boundary
Offer the product owners feasible options: keep affected operations unavailable, restore a known-good primary path, or apply a measured limited-path correction through the authorized network operator. Explain which tenants or products each option helps or risks. Avoid a universal MTU setting; gateway and tunnel topology affect the correct value.
When one product’s recovery harms another, the accountable portfolio or business owner chooses within delegated risk authority, with engineering and security input. If that authority is unavailable, continue approved containment and escalate with a decision deadline. Record the consequence of delay; silence is not approval.
A network workaround must not silently trigger database failover. If writers move between sites, separately establish fencing, replication lag, source of truth, and accepted recovery point. Application availability, transport recovery, and data consistency are distinct acceptance checks.
5. Assign permanent ownership and rehearse the handoff
After recovery, engineering leadership names the owner of the end-to-end service contract and the component owners. The network team may own tunnel configuration, the application team the representative synthetic, and the product team the degraded-service contract. One accountable problem owner integrates the acceptance evidence and escalates dependencies.
Exercise both failover and failback with the actual origin, authentication, and payload class. Add an escalation rehearsal with a missing primary responder. Document support hours, privileged-change authority, vendor contacts, and evidence exchange requirements. “Shared ownership” without an accountable escalation owner recreates the original gap.
Carry the work through the incident lifecycle
Recovery, investigation, instrumentation, and implementation can overlap. Keep service recovery and the underlying problem’s verification as separate states; reopen incident coordination if customer harm returns.
| Stage | Required work and closure evidence |
|---|---|
| Incident | Assign temporary command and boundary workstreams; log unreachable teams and test blind spots. |
| Remediation | Apply an authorized limited-path change or known-good route and verify both directions. |
| RCA | Analyze the technical failure together with contract gaps, access limits, vendor delays, and escalation authority. |
| Early warning | Monitor the real cross-boundary transaction, not only routing sessions or pod health. |
| Fixes in progress | Track each team’s change under one accountable problem owner, including vendor and security dependencies. |
| Tracking and verification | Require joint failover/failback evidence and an acknowledged operating handoff. |
| Retiring temporary alerts | End bridge-only per-path watches after permanent synthetic ownership and response are proven; retain the end-to-end service signal. |
Technical references
Use the documentation for your deployed version and actual topology. Scenarios are illustrative; proposed checks and thresholds require an approved exercise plan.