Partial Failure
In distributed systems, 'it worked' and 'it failed' aren't the only outcomes.
By the end of this lesson, handle the ambiguous 'did it happen?' outcomes that only exist in distributed systems.
In a single process, a function call succeeds or throws. Across a network, there is a third outcome: you sent a request, and you don't know what happened. A timeout doesn't tell you whether the server ignored the request, processed it and the reply was lost, or is still working on it. This ambiguity is the defining difficulty of distributed systems.
First, see the whole system. Then we’ll open it up.
The unknown outcome
- A client sends a request; the reply may be lost or delayed; on timeout the client cannot tell whether the server processed it.
flowchart LR client[Client] -->|request| server[Server] server -->|reply lost?| x[?] client -->|timeout| unknown[Unknown: did it run?]
You cannot eliminate the ambiguity, so you design around it: make operations idempotent so retrying an unknown outcome is safe, use idempotency keys so the server dedupes, prefer sagas with compensations over cross-service transactions, and record intent so you can reconcile later. 'Exactly once' is a fiction; 'at least once + idempotent' is the real pattern.
This is where the abstraction starts leaking.
RPC frameworks make a remote call look like a local one. The abstraction leaks exactly here: a local call can't half-happen, but every remote call can. Treating network calls as if they were local is the root of most distributed data bugs.
Ambiguous partial failure
- Trigger
- A network timeout leaves the caller unsure whether the operation completed.
- Symptom
- Double-charges, duplicate records, or lost work from mishandled retries.
- Blast radius
- Data integrity across the services involved in the operation.
- Mitigation
- Idempotency keys, sagas with compensation, reconciliation jobs, recorded intent.
Why is 'exactly once' delivery considered a fiction?
Networks can always lose the acknowledgment, so a sender can't know if a message arrived. The practical equivalent is at-least-once delivery with idempotent processing.