Rate Limiting
Bound how much each caller can do, protecting shared resources and enforcing fairness.
By the end of this lesson, choose a rate-limiting algorithm and reason about its bursts and fairness.
Rate limiting caps how many requests a caller may make in a window. It protects shared resources from abuse and accidental overload, enforces fairness across tenants, and is a first line of defense against retry storms and thundering herds.
Token bucket
- Tokens refill at a fixed rate into a bucket of capacity N; each request consumes a token; requests are allowed while tokens remain, otherwise rejected with 429.
flowchart LR refill[Refill tokens at fixed rate] --> bucket[(Bucket, capacity N)] req[Request] -->|take 1 token| bucket bucket -->|token available| allow[Allow] bucket -->|empty| deny[429]
Algorithms differ in how they treat bursts. Token bucket allows bursts up to the bucket size then settles to the refill rate. Leaky bucket enforces a smooth constant rate. Fixed window is simple but allows double-rate bursts at the window boundary. Sliding window smooths that boundary effect. Distributed rate limiting needs a shared counter (e.g., Redis) and must handle its own failure gracefully.
Precise global limits require a shared, consistent counter — a latency and availability dependency on every request. Approximate local limits are fast and resilient but let callers exceed the global cap. Most systems accept slight over-admission for speed.
Rate limiter as a single point of failure
- Trigger
- A centralized limiter store (e.g., Redis) becomes slow or unavailable.
- Symptom
- Every request blocks on the limiter, or the limiter is bypassed entirely.
- Blast radius
- All traffic gated by the limiter.
- Mitigation
- Fail-open vs fail-closed by design, local fallback limits, timeouts on the limiter call.
Deep dive: Dependency Exhaustion
Which algorithm allows short bursts but enforces an average rate?
Token bucket — it permits bursts up to the bucket capacity, then limits to the token refill rate.