Rate Limiting: Controlling the Flood
Rate limiting caps how many requests a client can make in a given window. It is the gate that keeps a single abusive user, a runaway script, or a sudden traffic spike from exhausting your capacity. Get it right and your service stays fair, affordable, and standing under load. Get it wrong and you either let attackers through or lock out legitimate users.
Why Rate Limit at All?
Every server has finite capacity. A rate limiter decides how much of that capacity any one client may consume, and rejects the rest. The reasons come down to four pressures.
Abuse & attacks
Brute-force logins, credential stuffing, scraping, and denial-of-service all rely on volume. A limiter blunts them by capping requests per client.
Fairness
One heavy user should not starve everyone else. Per-client limits give every tenant a fair slice of shared capacity.
Cost
Compute, bandwidth, and paid third-party APIs all cost money. A limiter puts a ceiling on spend and keeps a buggy client from running up the bill.
Protecting downstream
Your database, cache, and internal services have limits too. Shedding load at the edge prevents an overload cascade that takes the whole system down.
The Core Question
How much traffic can I safely accept per client? Rate limiting answers it. Below the limit, requests flow through. Above it, they are throttled — usually with an HTTP 429 Too Many Requests and a hint about when to try again.
Token Bucket and Leaky Bucket
The two bucket algorithms are the workhorses of rate limiting. Both use a fixed-size bucket, but they differ in whether they allow bursts.
Token Bucket
Tokens are added to a bucket at a steady refill rate, up to a maximum capacity. Each request removes one token. If the bucket is empty, the request is rejected. Because tokens accumulate up to capacity, a client that has been quiet can burst up to the full bucket size at once.
capacity = 10 tokens refill = 1 token / 100ms idle for 1s → bucket refills to 10 burst of 10 → all allowed (bucket now 0) 11th request → 429 (no tokens left) after 100ms → 1 token → 1 request allowed
Leaky Bucket
Requests enter a queue (the bucket) and are processed — "leak out" — at a constant rate. If the bucket is full, new requests overflow and are dropped. The output is perfectly smooth: no matter how bursty the input, downstream sees a steady stream.
bucket size = 10 requests leak rate = 1 request / 100ms burst of 10 arrives → queued processed at exactly 1 per 100ms 11th arrives while full → dropped Output is always ≤ 1 request / 100ms.
The Key Difference
Token bucket tolerates bursts — quiet time earns you credit to spend later. Leaky bucket forbids them — output is always a flat line. Choose token bucket for a good user experience with occasional spikes; choose leaky bucket when a fragile downstream must never be hit faster than its fixed rate.
Counting Windows: Fixed and Sliding
The other family of algorithms counts requests inside a time window. They range from dead-simple to precise-but-costly.
Fixed Window Counter
Divide time into fixed windows (e.g. one minute) and keep a counter per client per window. Increment on each request; reject once it exceeds the limit; reset the counter when the next window starts. Just one integer per client.
limit = 100 / minute 12:00:00–12:00:59 → count requests, reset at :00 key = "user:123:1200" → INCR, reject if > 100
The Boundary-Burst Problem
Fixed windows count each window independently, so a client can fire a full limit at the very end of one window and another full limit at the very start of the next — double the intended rate across a window-length span.
limit = 100 / minute 12:00:59 → 100 requests (allowed, window 12:00) 12:01:01 → 100 requests (allowed, window 12:01) 200 requests in ~2 seconds — 2x the limit, even though each window individually obeyed it.
Sliding Window Log
Store a timestamp for every request. On each new request, drop timestamps older than the window and count what remains. It is exact — the limit holds for any rolling window, killing the boundary problem — but you pay in memory: one entry per request.
log = [12:00:58, 12:00:59, 12:01:00, ...] new request at 12:01:30, window = 60s drop everything before 12:00:30 count remaining → allow if < limit, else 429
Sliding Window Counter
A practical middle ground. Keep counters for the current and previous fixed windows, then weight the previous window by how much of it still overlaps the rolling window. Nearly as smooth as the log, but with fixed O(1) memory — which is why it is the common production choice.
limit = 100 / minute
30% into current window (12:01)
estimate = current_count
+ previous_count * (1 - 0.30)
= 18 + 84 * 0.70
= 76.8 → under 100, allowWhere to Enforce the Limit
Choosing an algorithm is half the job. The other half is deciding where it runs and what it keys on.
At the API Gateway / Edge
The best place is as early as possible — the API gateway, load balancer, or CDN edge. Rejecting a request there means it never touches your application code or database, so the cheapest requests to serve are the ones you block. Gateways like Kong, NGINX, Envoy, and cloud API gateways all ship with built-in limiters.
What to Key On: Per-User vs Per-IP
Per-user (API key / account)
The fairest unit for authenticated traffic. Each account gets its own budget regardless of how many IPs it uses. The natural choice for tiered plans (free vs paid).
Per-IP
The only option before login (signup, password reset). But IPs are blunt: a corporate NAT or mobile carrier shares one IP across thousands of users, so an aggressive limit punishes the innocent.
Layered
In practice, combine them: a loose global per-IP limit to catch raw floods, plus a tighter per-user limit for fairness once the caller is identified.
Distributed Rate Limiting
With many gateway instances, a limiter that only counts locally lets a client multiply their allowance by the number of servers. To enforce one global limit, the counter must live in shared state — usually Redis, with atomic INCR and a script (Lua) to keep the check-and-decrement race-free.
# Redis fixed window, atomic key = "rl:user:123:1200" count = INCR key if count == 1: EXPIRE key 60 if count > limit: reject (429) # Sliding window log: ZADD timestamp, # ZREMRANGEBYSCORE old, ZCARD to count.
The trade-off: a network hop to Redis on every request. High-scale systems soften it by counting locally and syncing approximate totals, accepting slight overshoot for lower latency.
Responding to a Throttled Client
When you reject a request, tell the client clearly and politely. Good rate-limit responses are self-documenting.
HTTP 429 with Retry-After
The standard rejection is 429 Too Many Requests. Pair it with headers that tell the client its budget and when it resets, so well-behaved clients can back off instead of hammering.
HTTP/1.1 429 Too Many Requests
Retry-After: 30
RateLimit-Limit: 100
RateLimit-Remaining: 0
RateLimit-Reset: 30
{ "error": "rate_limited",
"message": "Try again in 30 seconds." }Client-Side Etiquette
A good client reads Retry-After and waits, then retries with exponential backoff and jitter so a thundering herd of clients does not all retry at the same instant. Surfacing RateLimit-Remaining also lets a client slow itself down before it hits the wall.
Live Token Bucket Simulation
Rate Limiting Pitfalls
- →Fixed windows without care: boundary bursts let clients briefly hit 2x the limit. Use a sliding window counter if that matters.
- →Counting only locally: with multiple servers, a per-instance limit multiplies by the server count. Use shared state (Redis) for a global limit.
- →Race conditions in the check: read-then-write without atomicity double-counts under concurrency. Use atomic INCR or a Lua script.
- →Per-IP as the only key: NATs and mobile carriers share one IP across many users, so aggressive IP limits punish the innocent.
- →No Retry-After: clients can't tell when to come back, so they retry immediately and make things worse.
- →Missing jitter on retries: synchronized backoff creates a thundering herd that spikes right when the window resets.
- →One global limit for everything: cheap reads and expensive writes shouldn't share a budget. Weight or bucket by cost.
- →Making the limiter a single point of failure: if Redis is down, decide up front whether to fail open (allow) or fail closed (block).
Key Takeaways
- →Rate limiting caps per-client traffic to protect against abuse, ensure fairness, control cost, and shield downstream services.
- →Token bucket allows bursts up to capacity; leaky bucket smooths output to a constant rate. Pick based on burst tolerance.
- →Fixed window is simplest but suffers boundary bursts. Sliding window log is exact but memory-heavy.
- →Sliding window counter approximates the log with O(1) memory and is the common production default.
- →Enforce as early as possible — the API gateway or edge — so blocked requests never reach your app.
- →Key on the user (API key) for fairness; fall back to per-IP before login, and layer both.
- →Across many instances, keep the counter in shared state (Redis) with atomic operations to enforce one global limit.
- →Reject with 429 plus Retry-After and RateLimit headers; good clients back off with exponential backoff and jitter.
Quick Reference: Algorithms Compared
Token Bucket
Allows bursts up to capacity, smooth average rate, O(1) memory. The go-to general-purpose choice.
Leaky Bucket
Constant output rate, no bursts, queues and smooths traffic. Best for protecting a fragile fixed-rate downstream.
Fixed Window Counter
One integer per client, dead simple. Vulnerable to boundary bursts of up to 2x the limit.
Sliding Window Log
Exact, no boundary bursts, but stores a timestamp per request — memory grows with volume.
Sliding Window Counter
Weighted blend of two fixed windows. Smooths boundaries with O(1) memory — the usual production pick.