Fault Tolerance: Timeouts, Circuit Breakers, Bulkheads, Backpressure and Graceful Degradation

At 9:02 p.m. the recommendation service, a non-essential widget at the bottom of Pixly's home page, starts responding in 8 seconds instead of 80 ms because of a bad deployment. By 9:05 the entire app is down. Uploads fail, feeds do not load, logins time out. The recommendation service never crashed, and nothing else had a bug.

What happened: every feed request waited 8 seconds on recommendations. With requests arriving at 3,500 per second, the API servers' threads all filled with waiting requests (Little's Law from article 02: 3,500 x 8 s = 28,000 requests in flight). New requests queued, then timed out at the load balancer. Clients retried. Load doubled. Health checks timed out, so the load balancer removed servers, pushing their traffic onto the rest, which collapsed faster.

This is a cascading failure, and it is how most large outages unfold. This article covers the techniques that contain failures so that a problem in one corner stays in that corner.

• The vocabulary: faults, failures, fault tolerance, cascading failure.
• How cascading failures happen, step by step.
• Timeouts and deadline propagation, and retry budgets.
• Circuit breakers, with a runnable simulation.
• Bulkheads and shuffle sharding for isolation.
• Rate limiting, load shedding and admission control under overload.
• Backpressure: making producers slow down.
• Graceful degradation and static stability, with a worked plan.
• How these mechanisms themselves fail, and how to test them.

• A fault is something going wrong in a component: a disk error, a bug, a slow network link.
• A failure is the system as a whole not doing its job for users.
• Fault tolerance is preventing faults from becoming failures.
• A cascading failure is a failure that spreads: one component's trouble overloads or starves others, which then fail too. Google's SRE book describes cascading failures as failures that grow over time through positive feedback, and the most common cause as overload (Addressing Cascading Failures).

1. A dependency becomes slow (or fails slowly)2. Callers wait; their threads, connections and memory fill up3. Callers become slow to their own callers -> the slowness moves upstream4. Timeouts fire; clients and services retry -> load increases5. Health checks time out -> load balancers remove servers -> survivors get more load6. Survivors exceed capacity and crash -> back to step 5, faster

Two features make it vicious. Positive feedback: each stage makes the next one worse. Slow is worse than down: a dependency that fails instantly costs callers almost nothing; one that hangs for 8 seconds costs them 8 seconds of resources per request.

Every technique below breaks one link in this chain.

Every remote call needs a timeout (article 18), so a slow dependency costs a bounded amount of time per request. Better still, propagate a deadline: the request arrives with "must finish by T", and each downstream call gets only the remaining time. A call made with 30 ms left should not wait 2 seconds.

In the opening scenario, a 200 ms timeout on recommendations would have limited each request's extra wait to 200 ms: painful but survivable.

Retries recover from transient faults but multiply load during real outages (article 20). Contain them:

• retry at one layer, a small number of times, with backoff and jitter;
• use a retry budget: allow retries only while they are a small fraction of total requests (for example 10%); beyond that, fail fast;
• never retry after the deadline has passed.

Even with timeouts, calling a dependency that is clearly down wastes a full timeout on every request and keeps hammering a service that is trying to recover. A circuit breaker sits in front of the call and stops calling a dependency that keeps failing, for a while.

Microsoft's description of the pattern uses three states (Circuit Breaker pattern):

• Closed: calls pass through; failures are counted. When failures exceed a threshold within a period, the breaker opens.
• Open: calls fail immediately, without touching the dependency, and the caller uses a fallback. After a timeout, the breaker becomes half-open.
• Half-open: a limited number of trial calls pass through. If they succeed, the breaker closes; if any fails, it opens again.

The same guidance notes that a circuit breaker serves a different purpose from retries: retries assume the operation will succeed soon; the breaker prevents calls that are likely to fail. Retry logic should stop when the breaker reports it is open.

This Java program simulates a breaker with a failure threshold of 3 and an open period of 5 seconds, using a fake clock so the output is repeatable. The dependency is down from 1 to 12 seconds.

// File: CircuitBreakerDemo.javapublic class CircuitBreakerDemo {    enum State { CLOSED, OPEN, HALF_OPEN }    static class CircuitBreaker {        private final int failureThreshold;        private final long openMillis;        private State state = State.CLOSED;        private int consecutiveFailures = 0;        private long openedAt = 0;        CircuitBreaker(int failureThreshold, long openMillis) {            this.failureThreshold = failureThreshold;            this.openMillis = openMillis;        }        String call(long now, boolean dependencyHealthy) {            if (state == State.OPEN) {                if (now - openedAt < openMillis) {                    return "rejected fast, fallback served";   // no call made                }                state = State.HALF_OPEN;                       // let one trial request through            }            boolean ok = dependencyHealthy;                    // the real remote call happens here            if (ok) {                consecutiveFailures = 0;                state = State.CLOSED;                return "success";            }            consecutiveFailures++;            if (state == State.HALF_OPEN || consecutiveFailures >= failureThreshold) {                state = State.OPEN;                openedAt = now;            }            return "failure";        }        State state() {            return state;        }    }    public static void main(String[] args) {        CircuitBreaker breaker = new CircuitBreaker(3, 5_000);        for (long t = 0; t <= 15_000; t += 1_000) {            boolean healthy = t < 1_000 || t >= 12_000;        // the dependency is down from 1 s to 12 s            String result = breaker.call(t, healthy);            System.out.printf("t=%2d s  dependency %-4s -> %-30s state %s%n",                    t / 1000, healthy ? "up" : "down", result, breaker.state());        }    }}

t= 0 s  dependency up   -> success                        state CLOSEDt= 1 s  dependency down -> failure                        state CLOSEDt= 2 s  dependency down -> failure                        state CLOSEDt= 3 s  dependency down -> failure                        state OPENt= 4 s  dependency down -> rejected fast, fallback served state OPENt= 5 s  dependency down -> rejected fast, fallback served state OPENt= 6 s  dependency down -> rejected fast, fallback served state OPENt= 7 s  dependency down -> rejected fast, fallback served state OPENt= 8 s  dependency down -> failure                        state OPENt= 9 s  dependency down -> rejected fast, fallback served state OPENt=10 s  dependency down -> rejected fast, fallback served state OPENt=11 s  dependency down -> rejected fast, fallback served state OPENt=12 s  dependency up   -> rejected fast, fallback served state OPENt=13 s  dependency up   -> success                        state CLOSEDt=14 s  dependency up   -> success                        state CLOSEDt=15 s  dependency up   -> success                        state CLOSED

Read it closely. After three failures the breaker opens, and for the next several seconds callers get an instant fallback instead of a timeout. At 8 seconds the half-open trial fails, so it reopens. At 12 seconds the dependency has recovered but the breaker is still open, so one more second passes before the trial at 13 seconds succeeds and traffic resumes. That short delay in recovery is the price of protecting the dependency from a flood the moment it comes back.

Production breakers (libraries such as Resilience4j in Java, or service-mesh features) usually use failure rates over a sliding window rather than consecutive counts, count slow calls as failures, and keep a breaker per dependency (and sometimes per shard or endpoint), so one bad partition does not block healthy ones.

On a ship, bulkheads divide the hull so one leak does not sink everything. In software, a bulkhead gives separate, limited resources to separate dependencies or workloads, so one cannot consume them all. Microsoft's Bulkhead pattern describes isolating elements into pools so that if one fails, the others continue to function (Bulkhead pattern).

Examples:

• Separate thread pools or concurrency limits per dependency. Calls to recommendations may use at most 50 concurrent slots; when they are full, further calls fail fast. The feed and profile calls have their own slots and are unaffected.
• Separate connection pools for the critical database path and for reporting queries.
• Separate queues and worker fleets for urgent and bulk work.
• Separate deployments per customer tier or per region.

You can size a bulkhead with Little's Law. If recommendations normally take 80 ms and receive 3,500 calls per second, about 3,500 x 0.08 = 280 calls are in flight; a limit of, say, 400 handles normal peaks, and in the 8-second scenario the excess fails fast instead of consuming every thread.

For multi-tenant systems, Amazon's Builders' Library describes shuffle sharding: each customer is assigned to a small, random combination of workers out of a larger fleet, so one customer's bad traffic (a "poison" request pattern) affects only the few workers in its shard, and almost no other customer shares exactly the same combination (Workload isolation using shuffle-sharding). It is an advanced form of bulkheading worth mentioning in SDE-2 discussions about noisy neighbours.

Protect the system from clients that send too much: per-user, per-API-key or per-IP limits, answered with 429 Too Many Requests. Microsoft's Throttling pattern frames this as controlling resource consumption so the system keeps meeting its service levels under heavy demand (Throttling pattern). Article 31 designs a distributed rate limiter.

Rate limits protect against individual clients. Sometimes the total load, from many well-behaved clients, exceeds capacity: a big event, a failover that moved traffic, a retry storm. A server that tries to serve everything then serves everything slowly, so almost every request times out, and useful throughput collapses. Google's SRE book discusses this, recommending that servers protect themselves by rejecting excess load early and cheaply (Handling Overload). The Amazon Builders' Library makes the same argument: shedding excess requests keeps latency low for the requests you accept, which keeps goodput high (Using load shedding to avoid overload).

How to shed:

• Measure saturation with something that reacts quickly: concurrent requests, queue length, or queue waiting time, rather than CPU alone.
• Reject early and cheaply: a fast 503 (or 429) before doing expensive work. A rejected request costs microseconds; a timed-out one costs seconds.
• Prioritise. Shed the least important traffic first: background prefetches, analytics events, non-critical widgets, before logins and purchases. Health checks and internal control traffic should be protected.
• Drop requests that have already waited too long. If a request has spent 3 seconds in a queue and the client times out at 2, processing it is pure waste.

In asynchronous pipelines, the equivalent of load shedding is backpressure: a slow consumer signals producers to slow down rather than letting buffers grow without limit.

• Bounded queues: when a queue is full, producers block, slow down, or receive an error, instead of memory growing until a crash.
• Pull-based consumption: consumers fetch work when they have capacity (most queue consumers already work this way).
• Propagate the signal upstream: an API whose work queue is too old returns 503 with Retry-After for new non-urgent jobs (article 11's backlog discussion).

Unbounded queues do not remove overload; they hide it until memory runs out or the work is too old to matter.

Graceful degradation means that when a dependency fails, the system continues with reduced functionality rather than failing entirely. It must be designed in advance, because nobody designs good fallbacks in the middle of an incident.

Common fallbacks:

Two supporting ideas:

• Kill switches and feature flags let operators turn off expensive or failing features instantly, without a deployment.
• Static stability: the Amazon Builders' Library describes designing systems so they keep operating in their current state when a dependency (such as a control plane) is impaired, for example by pre-provisioning enough capacity in each availability zone so that losing one zone does not require launching new resources during the emergency (Static stability using Availability Zones). The general lesson: avoid recovery plans that depend on the very systems most likely to be impaired during the incident.

First list the dependencies of the home-page request and decide, in advance, what happens when each fails.

Edge protections:

• Rate limits per user and per IP at the API gateway.
• Load shedding with three priority classes: (1) login, upload and other user actions; (2) feed loads; (3) prefetches, analytics events and recommendations. Under overload, class 3 is rejected first.
• Retry policy: the mobile app retries idempotent reads with backoff and jitter, once; services do not retry each other.

Replaying the 9:02 incident with these in place: recommendation calls start timing out at 200 ms; the breaker opens after a few seconds; the widget disappears; every other part of the page loads normally. Users notice a missing widget, not an outage, and the on-call engineer is paged by the breaker-open alert rather than by a total failure.

These mechanisms only prove themselves under failure, so create failure on purpose: inject latency and errors into dependencies in staging, run game days where a team deliberately disables a component and watches the system's response, and run load tests above expected peak to see where shedding begins. This practice is often called chaos engineering. The goal is not drama; it is discovering a broken fallback on a Tuesday afternoon rather than during the biggest sale of the year.

• No timeouts, or timeouts longer than the user's patience.
• Retries at every layer without budgets.
• Treating every dependency as critical, so any failure is a total failure.
• Fallbacks designed during the incident.
• Unbounded queues "to absorb spikes".
• Health checks that fail when a dependency fails (article 08).
• Never testing failure paths.

• For each dependency on the critical path, state the timeout and what happens when it fails.
• Mention circuit breakers for unreliable dependencies, with a fallback.
• Mention bulkheads when one slow dependency could consume shared resources.
• Under "what if traffic is 10 times normal?", talk about autoscaling limits, rate limiting and prioritised load shedding.
• Show a degradation table for the main page or flow. Interviewers rarely see one, and it signals production experience.

• Next: High Availability and Disaster Recovery: Redundancy, Failover, RPO, RTO and Multi-Region widens the view from one dependency to whole zones and regions.
• Then: Observability shows how to see these failures as they happen.