Site reliability engineering · Production operations
How to contain a failing downstream API
Protect a production service when a downstream API is slow, inconsistent, or returning incomplete results. Control how much work crosses the dependency boundary and define what happens to work already accepted.
1. Define semantic failure and preserve transaction identity
Start with the business operation, not the HTTP status. A successful transport response that omits an entitlement, price, or durable order reference may be unusable. Capture the schema expectation, affected API operation, correlation ID, caller version, and a sanitized example of the missing fields.
Separate transient transport errors, throttling, permanent validation errors, and an unknown write outcome. If a request times out after the remote system may have committed it, a retry can duplicate the action. Keep a stable operation ID and use supported idempotency or a reconciliation lookup before repeating a side effect. An idempotency key is useful only if its scope, retention, and concurrent behavior match the business operation.
2. Budget deadlines, retries, and concurrency together
Give the overall journey a deadline and leave time to return a meaningful result or reconcile. Propagate cancellation where supported. Allocate each dependency call less than the remaining deadline; increasing every timeout can turn a brief dependency failure into an exhausted caller pool.
Assign retry responsibility at one deliberate layer. Three attempts at each of three independently retrying layers can cause 27 leaf attempts from one initial request. Exponential backoff and jitter reduce synchronized pressure but do not add capacity. Honor supported server retry guidance within the remaining deadline and a bounded retry budget; do not retry permanent semantic errors merely because they arrived as HTTP 200.
Limit in-flight calls and queue length per relevant workload boundary. Bound queue age as well: a short queue of expensive jobs can already violate the business deadline. Apply admission control before memory and worker pools fill. Account for autoscaling: doubling pods can double downstream concurrency unless the limit is coordinated across the fleet.
3. Get product agreement on a safe degraded mode
Describe concrete alternatives to the product owner: serve a clearly identified stale recommendation, reject a new request quickly, or accept work durably for later processing. The last choice needs explicit pending semantics, bounded retention, a status path, and an owner for reconciliation. Returning “completed” for queued work misrepresents the user outcome.
Authorization, financial settlement, and regulatory checks cannot default to a guessed success. Route the choice through the service owner and the product/business authority, with security or compliance participation where required. If the owner is unreachable, use the approved existing degraded mode or stop the affected operation within delegated authority; do not invent a new business rule on the bridge.
Illustrative scenario: enterprise order creation depends on a partner’s credit-check API. Optional catalog enrichment may be disabled independently. New orders requiring a valid credit decision stay rejected or explicitly pending under the agreed contract; the application must not reuse another tenant’s cached decision.
4. Work through restricted diagnostics and vendor delays
When InfoSec prohibits response-body exports, ask for a schema-validity aggregate, selected field-presence checks, a redacted trace, and a correlation ID the vendor can locate. Limit evidence to the approved boundary and retention. Lack of packet capture access does not prevent measuring attempt counts, completion latency, queue age, and unknown transaction outcomes.
Open the vendor case with timestamps, region, operation, sanitized identifiers, and the expected versus actual contract. Use the contracted escalation route if the named contact is absent. A vendor’s request for broad administrator credentials is a new access request, not an obligation to comply during an outage. Keep containment active while the support case remains unresolved.
A circuit breaker needs a bounded half-open probe rate and a fallback with its own capacity limit. Otherwise every pod can probe at once, or the fallback can become the next failed dependency. Coordinate recovery of retries, queue drains, and autoscaling; lifting every limit simultaneously recreates the incident.
5. Recover queued work and verify correctness
Restore traffic gradually while observing original user operations, dependency attempts, queue age, and duplicate or missing outcomes. Drain oldest still-valid work within downstream capacity; expire or reconcile work whose business deadline passed. A healthy dependency graph does not clear a backlog automatically.
Keep the problem record open for a contract fix, client-library correction, or capacity change. Track temporary concurrency limits and degraded behavior as owned configurations. Product should explicitly accept the return to normal behavior; support needs a list of affected customers and an approved reconciliation process.
Carry the work through the incident lifecycle
Recovery, investigation, instrumentation, and implementation can overlap. Keep service recovery and the underlying problem’s verification as separate states; reopen incident coordination if customer harm returns.
| Stage | Required work and closure evidence |
|---|---|
| Incident | Identify semantic failure, write uncertainty, and the operations or tenants affected. |
| Remediation | Bound calls and retries, apply an approved degraded mode, and track already accepted work. |
| RCA | Reconstruct retry layers, deadline propagation, contract gaps, and dependency-capacity assumptions. |
| Early warning | Monitor completion correctness, retry amplification, queue age, and unknown transaction outcomes. |
| Fixes in progress | Keep temporary limits and reconciliation staffed while client or vendor corrections are pending. |
| Tracking and verification | Test timeouts after remote commit, half-open recovery, and queue drain; attach results to the shared problem and vendor case. |
| Retiring temporary alerts | Retire incident-specific queue or vendor watches after reconciliation and permanent boundary coverage pass; retain the user-outcome and dependency alerts. |
Technical references
Use the documentation for your deployed version and actual topology. Scenarios are illustrative; proposed checks and thresholds require an approved exercise plan.