Flat Spell Technologies

Site reliability engineering · Production operations

How to resolve cross-team ownership during an incident

Coordinate a shared failure without waiting for an organizational dispute to be settled. Establish temporary response authority, evidence at the boundary, and a lasting owner for the work that remains after recovery.

1. Separate incident coordination from service ownership

Have the duty engineering authority name a temporary incident commander and technical coordinator. Record the assignment and escalation route. The coordinator allocates investigation and communicates the next decision; each production action still belongs to an authorized operator for that system.

Create workstreams around boundaries: user-to-edge, edge-to-application, application-to-data, and hybrid transit. Give each workstream a named responder, the question to answer, and a next update time. A request to “look at your side” creates parallel narratives rather than a common test. Permanent service ownership can be decided after containment, but temporary task ownership cannot remain implicit.

2. Make boundary evidence comparable

Illustrative scenario: on-premises clients reach a private Azure application over ExpressRoute, with an IPsec VPN backup. After failover, small checks pass but large authenticated responses stall. Network operations sees established routing sessions; application operations sees healthy pods. Both can be correct while the customer journey fails.

Agree a timestamped test matrix: origin, destination, identity class, payload size, direction, active route, and observed outcome. Run the same permitted transaction from cloud and on premises. Compare edge captures or approved flow evidence, effective MTU, TCP MSS, and path-MTU feedback. Test asymmetric routing and firewall state as alternatives before declaring an MTU cause.

A routing session proves control-plane adjacency, not complete payload delivery. The operator controlling each edge should collect only the necessary diagnostic evidence; a packet capture can expose regulated payloads. Use metadata, filtered captures, or synthetic payloads where policy requires them.

3. Work around missing teams without crossing their authority

If the network primary does not acknowledge, use the secondary, duty manager, and contracted provider escalation. Keep the response record explicit about which edge cannot yet be observed. Ask the available team for tests that discriminate without changing that edge, such as size-controlled synthetic transactions or approved flow logs.

If InfoSec will not allow a broad firewall change or packet capture, propose a narrower filter, an approved capture by its operator, or a test environment reproducing the path. Record what uncertainty remains. Do not solve an ownership dispute by issuing a global security exception or disabling inspection controls.

When external support is delayed, give the vendor the shared test matrix and an owner for updates. A vendor case is a dependency in the recovery plan, not a transfer of the incident. Keep customer communications and available containment under internal ownership.

4. Escalate the business decision at the shared boundary

Offer the product owners feasible options: keep affected operations unavailable, restore a known-good primary path, or apply a measured limited-path correction through the authorized network operator. Explain which tenants or products each option helps or risks. Avoid a universal MTU setting; gateway and tunnel topology affect the correct value.

When one product’s recovery harms another, the accountable portfolio or business owner chooses within delegated risk authority, with engineering and security input. If that authority is unavailable, continue approved containment and escalate with a decision deadline. Record the consequence of delay; silence is not approval.

A network workaround must not silently trigger database failover. If writers move between sites, separately establish fencing, replication lag, source of truth, and accepted recovery point. Application availability, transport recovery, and data consistency are distinct acceptance checks.

5. Assign permanent ownership and rehearse the handoff

After recovery, engineering leadership names the owner of the end-to-end service contract and the component owners. The network team may own tunnel configuration, the application team the representative synthetic, and the product team the degraded-service contract. One accountable problem owner integrates the acceptance evidence and escalates dependencies.

Exercise both failover and failback with the actual origin, authentication, and payload class. Add an escalation rehearsal with a missing primary responder. Document support hours, privileged-change authority, vendor contacts, and evidence exchange requirements. “Shared ownership” without an accountable escalation owner recreates the original gap.

Carry the work through the incident lifecycle

Recovery, investigation, instrumentation, and implementation can overlap. Keep service recovery and the underlying problem’s verification as separate states; reopen incident coordination if customer harm returns.

Lifecycle handoff for this workstream
StageRequired work and closure evidence
IncidentAssign temporary command and boundary workstreams; log unreachable teams and test blind spots.
RemediationApply an authorized limited-path change or known-good route and verify both directions.
RCAAnalyze the technical failure together with contract gaps, access limits, vendor delays, and escalation authority.
Early warningMonitor the real cross-boundary transaction, not only routing sessions or pod health.
Fixes in progressTrack each team’s change under one accountable problem owner, including vendor and security dependencies.
Tracking and verificationRequire joint failover/failback evidence and an acknowledged operating handoff.
Retiring temporary alertsEnd bridge-only per-path watches after permanent synthetic ownership and response are proven; retain the end-to-end service signal.

Technical references

Use the documentation for your deployed version and actual topology. Scenarios are illustrative; proposed checks and thresholds require an approved exercise plan.