Clock Skew
When machines disagree about the time, tokens, ordering, and caches break subtly.
By the end of this lesson, anticipate where time assumptions break in distributed systems.
Distributed systems love to assume a shared clock. They don't have one. Physical clocks drift; even with NTP they disagree by milliseconds to seconds. That skew breaks anything that compares timestamps across machines: token validity, event ordering, cache expiry, certificate checks, and lock leases.
Two clocks, one decision
- An issuer and verifier with skewed clocks disagree on whether a token is valid, causing spurious rejections or acceptance of expired tokens.
flowchart LR issuer[Issuer clock: 10:00:05] --> token[Token exp=10:00:00] verifier[Verifier clock: 09:59:58] --> reject[Rejects 'not yet valid' / accepts expired]
Defenses: sync clocks with NTP and monitor drift; allow a small skew tolerance on exp/nbf (a few minutes); never use wall-clock timestamps to order events across nodes — use logical clocks (Lamport) or version vectors; and for leases/locks use fencing tokens rather than trusting time alone.
Clock skew
- Trigger
- Machines' clocks drift and a decision compares timestamps across them.
- Symptom
- Valid tokens rejected or expired ones accepted; mis-ordered events; premature lease expiry.
- Blast radius
- Auth failures, subtle data ordering bugs, and lock/lease errors across services.
- Mitigation
- NTP + drift monitoring, skew tolerance on token times, logical clocks for ordering, fencing tokens.
Why not order distributed events by their wall-clock timestamps?
Clocks disagree, so wall-clock order can be wrong. Logical clocks or version vectors capture causal order without a shared clock.