Retry with Backoff
Retry transient failures — but with backoff, jitter, and a budget, or you build a retry storm.
By the end of this lesson, design retries that improve reliability without amplifying failures.
Many failures are transient — a blip, a brief timeout, a momentary overload. Retrying often succeeds. But naive retries (immediate, unlimited, synchronized) turn a small failure into a retry storm. The pattern is retry *carefully*: only transient errors, with growing waits, randomized, and capped.
Exponential backoff with jitter
- Each retry waits exponentially longer with random jitter, up to a cap, then gives up.
flowchart LR a[Attempt 1] -->|fail, wait ~1s +/- jitter| b[Attempt 2] b -->|fail, wait ~2s +/- jitter| c[Attempt 3] c -->|fail, wait ~4s| stop[Give up / DLQ]
Rules: exponential backoff (1s, 2s, 4s…) so you don't hammer; jitter so clients don't re-synchronize; a retry budget (cap retries as a percentage of traffic) so retries can't multiply load without bound; retry only idempotent operations; and never retry a permanent error (400, 401, 403). Combine with circuit breakers to stop retrying a dead dependency.
Retry amplification
- Trigger
- Retries without backoff/jitter/budget against a degraded dependency.
- Symptom
- Multiplied load prevents recovery — a retry storm.
- Blast radius
- The dependency and everything queued behind it.
- Mitigation
- Exponential backoff, jitter, retry budgets, idempotency, circuit breakers.
Deep dive: Retry Storms
Should you retry a 403 Forbidden?
No — it's a permanent authorization failure that will fail identically every time. Retry only transient errors.