Split Brain
A network partition leaves two halves of a cluster each believing it's in charge.
By the end of this lesson, explain how partitions cause divergence and how quorum and leases prevent split brain.
Split brain happens when a network partition cuts a cluster in two, and each side, unable to see the other, assumes the other is dead and promotes itself. Now two nodes accept writes as 'the leader'. When the partition heals, their states conflict — and some writes must be lost.
First, see the whole system. Then we’ll open it up.
Partition -> two leaders
- A network cut separates nodes A and B; each declares itself leader and accepts writes, which will conflict when the partition heals.
flowchart TB
subgraph left [Partition A]
a[Node A: 'I am leader']
end
subgraph right [Partition B]
b[Node B: 'I am leader']
end
a -. network cut .- b
a --> wa[Accepts writes]
b --> wb[Accepts conflicting writes]Prevention rests on quorum: require a majority (more than half) to elect a leader or commit a write. A minority partition cannot reach majority, so it steps down rather than accepting writes. Fencing tokens (monotonic epoch numbers) ensure a deposed old leader's late writes are rejected. This is why consensus systems need an odd number of nodes.
This is where the abstraction starts leaking.
CAP is not academic here: during a partition you choose consistency (the minority refuses writes, reducing availability) or availability (both sides serve, risking split brain). 'Both, always' is not on the menu.
Split brain
- Trigger
- A network partition with a failover mechanism that lacks quorum or fencing.
- Symptom
- Two nodes both act as leader and accept divergent writes.
- Blast radius
- Data divergence and lost writes across the whole dataset.
- Mitigation
- Quorum-based election, fencing tokens, odd node counts, consistency-favoring failover.
Why do consensus clusters use an odd number of nodes?
So a majority (quorum) can always be formed on exactly one side of a partition, preventing two leaders.