The instinct when an LLM feature slows under load is to add application servers. It rarely helps, because your servers are idle — they spend almost all their time waiting on the model API. The true constraint is the provider's rate limit, chiefly tokens per minute. Scaling an LLM app is the art of doing more useful work within a fixed token budget.
Know your real ceiling
Your throughput ceiling is roughly the provider's TPM divided by the average tokens per request. Halve the tokens per request and you double your capacity without touching infrastructure. That reframes scaling: trimming a bloated system prompt or capping retrieved context is a throughput improvement, not just a cost one.
Queue and prioritise
Not all work is equally urgent. Put background jobs — enrichment, batch summarisation, offline classification — behind a queue with a bounded worker pool, and keep foreground user requests on a fast path. When load spikes, the queue absorbs it and drains a little slower; the user-facing latency stays flat. Without this, one batch job can consume the whole token budget and stall live traffic.
Reduce tokens before adding capacity
- Cache aggressively — every cache hit is throughput you got for free, and provider prompt caching stretches TPM further on long prompts.
- Batch where the provider supports it, and consolidate several small classifications into one call when latency allows.
- Right-size the model — a smaller, cheaper model for routine calls leaves the flagship's budget for the calls that need it.
- Trim context — shorter prompts and fewer retrieved chunks are a direct throughput multiplier.
Spread the load
When you have genuinely outgrown a single account's limits, distribute across multiple keys, projects, or regions, and request higher tier limits ahead of the demand. A gateway makes this transparent — it round-robins or shards across keys while the rest of the app calls one internal endpoint.
Every system has a maximum sustainable rate. When you hit it, the humane failure is a queue with a bounded wait or an honest 'try again shortly' — not accepting unlimited work and letting latency climb until everything times out. Backpressure keeps a busy system responsive instead of collapsing.
Scale in this order: cut tokens per request, cache, queue and prioritise, right-size models, and only then spread across more capacity. Almost every LLM scaling win comes from using the token budget more efficiently, not from bigger machines.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.