Rate Limiting LLM Apps

Rate limiting in an LLM app is two problems wearing one name. There is the provider's limit you must stay under — usually expressed as requests per minute and tokens per minute — and there is your limit, the per-user quota that stops one customer or one abusive loop from consuming the whole account. You need both.

Staying under the provider ceiling

Providers enforce two independent budgets: requests per minute (RPM) and tokens per minute (TPM). You can be well under RPM and still get throttled because a few large prompts blew the TPM budget. When you exceed either, the API returns HTTP 429, usually with a retry-after header telling you how long to wait. The single most important rule: respect retry-after. Retrying immediately just deepens the hole.

import time, random

def with_backoff(fn, max_tries=5):
    for attempt in range(max_tries):
        try:
            return fn()
        except RateLimitError as e:
            if attempt == max_tries - 1:
                raise
            wait = e.retry_after or (2 ** attempt + random.random())
            time.sleep(wait)

The jitter matters. Without the random component, every worker that got throttled at the same instant retries at the same instant, and you re-trigger the limit in a thundering herd. Exponential backoff plus jitter spreads the retries out.

Shape traffic with a queue

Backoff is a reaction. For steady load you want to avoid the 429 in the first place by metering outbound calls yourself with a token-bucket limiter sized just under your provider ceiling. Put bursty work — batch jobs, background enrichment — behind a queue with a fixed pool of workers, so concurrency is bounded by design. Foreground user requests get priority; background work fills the slack. This turns "we hit the limit and everything failed" into "the queue drained a little slower."

Per-user quotas

Your own limits protect the account and your margins. Enforce a per-user or per-tenant budget — requests, tokens, or cost — in the request path, and when a user exceeds it, return a clear error or a degraded response rather than silently passing the cost through. Track it in the same store you use for cost monitoring; the two are the same data viewed two ways.

Done well, rate limiting is invisible: users never see a 429, background jobs run at a sustainable pace, and no single tenant can starve the rest. Done badly, it is the outage that only happens under load — which is exactly when you least want to be debugging it.