Retry Storms
A struggling service gets hammered by everyone's retries and can never recover.
By the end of this lesson, explain how naive retries amplify load and design retries that don't make outages worse.
A service starts failing. Every client retries. The retries multiply the load on the already-struggling service, guaranteeing it stays down. Retries meant to improve reliability instead prevent recovery. This is a retry storm — a positive feedback loop.
Retry amplification
- A degraded service triggers clients to retry (e.g., 3x), tripling load, causing more failures, causing more retries — a loop.
flowchart LR fail[Service degraded] --> r1[Clients retry 3x] r1 --> load[3x load] load --> worse[More failures] worse --> r1
Fixes: exponential backoff (wait longer each attempt), jitter (randomize waits so clients don't synchronize), a retry budget (cap retries as a fraction of traffic), and circuit breakers (stop calling a dead dependency entirely). Also: only retry idempotent operations, and never retry on a 4xx that will always fail.
This is where the abstraction starts leaking.
'Just add retries' is where retry storms are born. Retries without backoff, jitter, and a budget are a load amplifier aimed at your weakest component at its worst moment.
Retry storm
- Trigger
- Clients retry a failing dependency without backoff/jitter/budget.
- Symptom
- Load spikes on the failing service, preventing recovery even after the root cause clears.
- Blast radius
- The dependency and everything queued behind it.
- Mitigation
- Exponential backoff with jitter, retry budgets, circuit breakers, idempotency.
Why add jitter to backoff?
Without jitter, clients that failed together retry together, re-synchronizing into repeated load spikes. Jitter spreads them out.