Foundation16 min

Distributed Systems

The moment your system spans more than one machine, new laws apply.

By the end of this lesson, reason about partial failure, asynchrony, and the fallacies that break naive distributed designs.

How deep?

How the pieces actually move.

A distributed system is one where components on separate machines coordinate over a network. That single fact changes everything: the network can drop, delay, duplicate, or reorder messages; any node can fail independently; and there is no shared clock or shared memory. Techniques that are trivial on one machine (a function call, a lock, a transaction) become hard, expensive, or impossible.

First, the whole system

First, see the whole system. Then we’ll open it up.

Nodes coordinating over an unreliable network

Nodes A, B, and C communicate only through an unreliable network that can drop, delay, or reorder messages; each node can fail independently.

flowchart LR
  a[Node A] <-->|may drop / delay / reorder| net((Network))
  net <--> b[Node B]
  net <--> c[Node C]
  a -.independent failure.- a

Abstraction leak

This is where the abstraction starts leaking.

The Fallacies of Distributed Computing name the lies we tell ourselves: the network is reliable, latency is zero, bandwidth is infinite, the network is secure, topology never changes, there's one administrator, transport cost is zero, the network is homogeneous. Every one of them is false, and every one causes an outage when assumed.

The defining problem is partial failure: some parts work while others don't, and you often can't tell the difference between a slow node and a dead one. A timeout doesn't tell you whether the request failed, succeeded, or is still running — which is exactly why idempotency and retries with backoff exist.

How it connects

The whole rest of this domain builds on these constraints:

Now break it

Node A calls Node B and the call times out. What are the three possible states of B's work, and why can't A tell them apart?

Cascading failure

Trigger: A slow dependency plus shared resources and no isolation.
Symptom: One failure drains pools and spreads until the whole system is down.
Blast radius: Potentially the entire system.
Mitigation: Timeouts, circuit breakers, bulkheads, backpressure.

Deep dive: Cascading Failure

Why is 'partial failure' the hardest part of distributed systems?

Because some components fail while others keep working, and you often can't distinguish a slow node from a dead one, so you must design for ambiguous outcomes.