API Throttling Patterns

Throttling and rate limiting are often used interchangeably, but the useful distinction is that rate limiting typically rejects requests outright once a threshold is crossed, while throttling slows them down — delaying, queuing, or degrading service gracefully rather than returning a hard error. Which patterns you reach for depends on whether you're protecting your own infrastructure or shaping traffic for someone else's benefit.

Token bucket vs. leaky bucket

Token bucket is the most common algorithm: a bucket holds tokens up to a cap, tokens refill at a fixed rate, and each request consumes one. This naturally allows bursts — a client that's been idle can fire a flurry of requests up to the bucket's capacity before being throttled — which suits most real traffic patterns, since legitimate clients are bursty rather than perfectly steady. Leaky bucket enforces the opposite property: requests go into a queue and drain out at a strictly constant rate, smoothing bursts into a steady stream regardless of how they arrived. Token bucket is the better default for API rate limiting because it tolerates legitimate burstiness; leaky bucket is the better fit when you need to protect a downstream system that genuinely cannot handle bursts at all, like a legacy database with a hard connection cap.

class TokenBucket:
    def __init__(self, capacity, refill_rate):
        self.capacity = capacity
        self.tokens = capacity
        self.refill_rate = refill_rate  # tokens per second
        self.last_check = time.monotonic()

    def allow(self):
        now = time.monotonic()
        elapsed = now - self.last_check
        self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_rate)
        self.last_check = now
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False

Sliding window: smoother limits at a memory cost

Fixed-window counters (reset the count every 60 seconds on the clock) are simple but have an edge case: a client can send a full window's worth of requests in the last second of one window and again in the first second of the next, doubling the effective rate right at the boundary. A sliding window — weighting the previous window's count by how much of it overlaps the current moment, or tracking individual request timestamps — avoids that burst-at-the-boundary problem at the cost of more state per client. For most APIs, a sliding window approximation (weighted average of the current and previous fixed windows) gets close enough accuracy without the cost of storing every timestamp.

Per-key throttling vs. global capacity protection

Throttling needs to happen at more than one granularity. Per-API-key or per-user limits enforce fairness and plan tiers — one noisy customer shouldn't be able to degrade service for everyone else. A global limit protects total system capacity regardless of how it's distributed across keys — useful during a traffic spike or a partial outage where even well-behaved, individually-compliant clients can collectively overwhelm a degraded backend. Systems that only implement per-key limits can still fall over if enough distinct keys hit the API simultaneously; you need both layers.

Graceful degradation instead of a flat rejection

A 429 response is the blunt tool. A more considerate throttling strategy degrades gracefully first: serve a cached or slightly stale response instead of a fresh computed one, drop optional fields from the response to reduce backend load, or queue the request with a delay rather than rejecting it outright. This matters most for internal APIs where you control both ends and can afford a slower response far more easily than a failed one — for public APIs, a clear 429 with a Retry-After header is usually the more honest contract.

Wrapping up

Start with a token bucket per API key plus a global capacity ceiling — it covers the overwhelming majority of real traffic shapes. Add sliding-window precision or graceful degradation only once you've observed a specific gap the simple version doesn't cover, and always give a throttled client a concrete number to retry against.