Why Distributed Systems Are Hard: Partial Failure, Timeouts and Clocks

Your service calls the payment provider to charge a customer ₹4,999. After two seconds, the call times out. What happened?

Maybe the request never reached the provider. Maybe it reached the provider, the card was charged, and the response was lost on the way back. Maybe the provider is still processing it and will charge the card in three seconds. From your side, all three look exactly the same: no response. If you retry, you might charge twice. If you do not, you might never charge at all.

On one machine, a function call either returns or throws. Across a network, there is a third outcome, "I don't know", and it changes everything. This article explains the handful of facts that make distributed systems hard, so that the techniques in the rest of Module 6 (consistency models, idempotency, sagas, locks) make sense as answers to real problems.

• What makes a system "distributed", and why you already have one.
• Partial failure, and why it is different from a crash on one machine.
• What networks do to messages: loss, delay, duplication and reordering.
• Why a timeout cannot tell you what happened, and how to choose timeouts.
• Process pauses and why a node can be wrong about its own state.
• Why clocks disagree, why wall-clock time cannot order events across machines, and how logical clocks help.
• Design principles that follow, used throughout the rest of the course.

A distributed system is a set of programs on different machines that cooperate by sending messages over a network. Pixly, as designed so far, already is one: clients, load balancers, a fleet of API servers, a cache cluster, a primary database with replicas, a queue and workers, object storage, a CDN, and third-party email and payment providers. Every arrow in the diagram is a network message that can go wrong.

Amazon's Builders' Library has an article on exactly this topic, arguing that the hard part of distributed programming is that every network interaction has many more ways to fail than a local call, and that engineers must handle each of them (Challenges with distributed systems).

On a single computer, failure tends to be total: if the machine crashes, everything on it stops together. In a distributed system, parts fail independently:

• one of eighteen API servers runs out of memory;
• the network link between two availability zones drops packets;
• the cache is healthy but slow because one node is swapping;
• the database primary is fine, but one replica is ten seconds behind;
• the email provider accepts requests but delivers them an hour late.

This is partial failure, and it is the defining difficulty. The system is neither up nor down; it is partly working, in ways that are hard to observe and that change from moment to moment. Code written for "it works or it throws" is not ready for "it half works, sometimes, slowly".

A network between two machines can:

TCP hides some of this inside one connection (it resends lost packets and keeps order), but not across connections, retries, queues or services. At the application level, assume all five happen.

There is also a deeper limit. Two parties communicating over a link that can lose messages cannot be certain they agree: every confirmation might itself be lost, and so might the confirmation of the confirmation. This is often explained as the "two generals" problem. Distributed systems therefore never achieve certainty by sending more messages; they use designs that stay correct when messages are lost or repeated.

When a request gets no reply within a deadline, you know only that you did not hear back. The possibilities:

1. The request was lost       -> nothing happened2. The work happened, but the reply was lost   -> it DID happen3. The work is still happening (slow server)   -> it MAY happen later

You cannot tell these apart from the caller's side. This one fact drives much of distributed-systems design:

• Retrying is necessary (case 1 is common), but dangerous (cases 2 and 3 make retries duplicates). Retries are only safe for idempotent operations, which is why article 20 is about idempotency.
• "Did my request succeed?" must be answerable later, for example by querying status with an idempotency key or request id.
• The callee must tolerate duplicates, because callers will retry.

Too short, and healthy-but-slow requests are cut off and retried, adding load exactly when the system is slow. Too long, and threads and connections are held hostage by a dead dependency (Little's Law, article 02).

A sound approach, described in the Amazon Builders' Library article on timeouts and retries, is to base the timeout on the downstream service's latency distribution: pick an acceptable rate of false timeouts (for example 0.1%) and set the timeout near the matching latency percentile (for example p99.9) (Timeouts, retries and backoff with jitter). Then check that your own overall deadline still meets your latency requirement: a request that calls three services in sequence cannot give each one a 2-second timeout and still promise a 1-second response.

Two more practical habits:

• Propagate deadlines. If the user's request has 800 ms left, a downstream call should not wait 2 seconds. Pass the remaining budget along.
• Separate connection and request timeouts. Failing to connect is usually quick to detect; a stuck response is not.

A program can stop running for a while without knowing it:

• a garbage-collection pause in a managed runtime such as the JVM;
• the virtual machine being paused or migrated by the hypervisor;
• the operating system swapping memory to disk;
• a laptop lid closing on a developer's machine running a "production" script.

When the program resumes, it continues from where it was, believing almost no time has passed. If it held a lock that expired during the pause, or was the leader and another node was elected meanwhile, it will now act on authority it no longer has. Martin Kleppmann's analysis of distributed locking walks through exactly this scenario, with long garbage-collection pauses, and shows why locks for correctness need fencing tokens that let storage reject stale holders (How to do distributed locking). Article 22 builds on this.

The general lesson: a node cannot fully trust its own sense of time or status. Correctness must be enforced where the data lives, not only by the node that believes it is in charge.

Every machine has a clock, and the clocks disagree.

• Skew and drift: hardware clocks run slightly fast or slow. Synchronisation protocols (such as NTP) correct them, but only to within some error, which depends on the network.
• Jumps: when a clock is corrected, it can jump forward or backward. A timestamp taken later can be smaller than one taken earlier.
• Two kinds of clock: a wall clock (time of day, suitable for showing users and for logs) and a monotonic clock (only moves forward, suitable for measuring durations and timeouts on one machine). Use the monotonic one for "how long did this take?".

Suppose server B records "post created" and sends it to server A, which then records "comment added". If A's clock runs 50 ms behind B's, the comment can carry an earlier timestamp than the post it replies to. Sorting by timestamp shows the reply before the post. Worse, a database using last write wins by timestamp can silently keep the older of two writes.

In 1978, Leslie Lamport showed how to order events consistently with cause and effect without synchronised clocks (Time, Clocks, and the Ordering of Events in a Distributed System). Each process keeps a counter:

1. Increment it before each local event.
2. Attach it to every message sent.
3. On receiving a message, set the counter to max(local, received) + 1.

If event X could have caused event Y (X happened before Y on the same process, or X's message was received before Y), then X's counter is smaller than Y's. This Lamport clock gives an order that respects causality.

The program below simulates the post-and-comment example: wall clocks with 50 ms of skew put the comment first; Lamport clocks put the post first.

// File: ClockDemo.javaimport java.util.ArrayList;import java.util.Comparator;import java.util.List;public class ClockDemo {    record Event(String what, String server, long wallClockMs, long lamport) {}    public static void main(String[] args) {        long skewOfA = -50;          // server A's wall clock runs 50 ms behind true time        long lamportA = 0;        long lamportB = 0;        List<Event> events = new ArrayList<>();        // True time 1000 ms: server B creates a post.        lamportB++;        events.add(new Event("post created", "B", 1000, lamportB));        long stampOnMessage = lamportB;            // B sends the post to A, with its Lamport time        // True time 1003 ms: A receives it. Rule: max(local, received) + 1.        lamportA = Math.max(lamportA, stampOnMessage) + 1;        // True time 1004 ms: A records a comment on that post.        lamportA++;        events.add(new Event("comment added", "A", 1004 + skewOfA, lamportA));        List<Event> byWallClock = new ArrayList<>(events);        byWallClock.sort(Comparator.comparingLong(Event::wallClockMs));        System.out.println("Sorted by wall clock:");        for (Event e : byWallClock) {            System.out.println("  " + e.wallClockMs() + " ms  " + e.what() + " (server " + e.server() + ")");        }        List<Event> byLamport = new ArrayList<>(events);        byLamport.sort(Comparator.comparingLong(Event::lamport));        System.out.println("Sorted by Lamport clock:");        for (Event e : byLamport) {            System.out.println("  L=" + e.lamport() + "  " + e.what() + " (server " + e.server() + ")");        }    }}

Sorted by wall clock:  954 ms  comment added (server A)  1000 ms  post created (server B)Sorted by Lamport clock:  L=1  post created (server B)  L=3  comment added (server A)

Lamport clocks have a limit: if two events are concurrent (neither could have caused the other), their counters still compare, so the order between them is arbitrary. Vector clocks (a counter per process) can detect that two updates are concurrent, which is useful for spotting conflicts. Amazon's Dynamo paper describes using vector clocks to detect conflicting versions of the same object (Dynamo).

Some systems do need globally meaningful timestamps. Google Spanner, for example, uses its TrueTime infrastructure, which exposes clock uncertainty explicitly and waits it out to give transactions a consistent global order (Spanner: TrueTime and external consistency). The lesson for ordinary designs is not to copy that, but to notice what it took: special hardware and careful protocol design. Without such infrastructure, do not rely on timestamps from different machines to decide which write wins.

These habits appear in every remaining article. Learn them as a list:

1. Assume every remote call can fail, hang, or succeed without telling you. Give it a timeout and a plan.
2. Make operations idempotent so retries and redeliveries are safe.
3. Prefer at-least-once delivery plus deduplication over hoping for exactly-once.
4. Do not order events by wall-clock time across machines. Use sequence numbers and versions.
5. Enforce correctness where the data lives (constraints, conditional writes, fencing), not only in the node that thinks it is in charge.
6. Keep the synchronous critical path short. Every hop is another way to fail.
7. Design for partial failure: degrade a feature instead of failing everything.
8. Make the system observable, because partial failures are invisible without good signals (article 25).
9. Prefer simple designs. Each extra component and protocol multiplies the failure combinations.

• Treating a timeout as "it failed" and blindly retrying a non-idempotent operation.
• No timeouts, or one fixed timeout for everything.
• Using System.currentTimeMillis() (or any wall clock) differences across machines for ordering or conflict resolution.
• Assuming messages arrive once and in order.
• Assuming a lock or a leadership role is still held because no error was seen.
• Testing only the happy path; never injecting delays, drops or duplicates.

• When a design calls another service, say what happens on timeout, and whether retries are safe.
• When ordering matters, say where the sequence number comes from.
• When something is replicated or partitioned, mention partial failure: "if the link between zones drops, this side will..."
• Use the vocabulary precisely: partial failure, idempotent, at-least-once, clock skew, fencing.

• Next: CAP Theorem and Consistency Models Explained Without the Myths asks what "consistent" can mean when replicas and partitions are involved.
• Then: idempotency and retries (article 20), distributed transactions (article 21), and locks and leader election (article 22) answer the problems listed here.