An LLM app is a web service, so most of deploying one is the usual work — stateless services, health checks, secrets, autoscaling. What is different is that calls are slow and streamed, metered by the token, and non-deterministic, and those three properties shape a handful of decisions worth getting right up front.
Put a gateway in front of the model
Do not scatter provider SDK calls across your codebase. Route every model call through one internal gateway — a thin service or module — that owns authentication, retries, fallbacks, rate limiting, caching, and usage logging. When you need to swap a model, add a provider, or change the retry policy, you change it once. This single choke point is also where cost tracking and prompt-version stamping live, so nothing slips past unlogged.
Stream responses
A full generation can take many seconds; making the user stare at a spinner for all of it is a bad experience and ties up a connection. Stream tokens as they arrive so the response starts rendering immediately. That means your infrastructure — load balancers, proxies, timeouts — must allow long-lived streaming connections, which is a common thing to forget until the first response gets cut off at 30 seconds.
Timeouts and statelessness
- Set explicit client timeouts on model calls — the default is often too long, and a hung upstream should fail fast into your fallback path.
- Keep the service stateless so you can autoscale horizontally; conversation state lives in a store, not in process memory.
- Hold provider keys in a secret manager, scope them, and rotate them — a leaked LLM key is directly billable to you.
# Config as environment / secrets — never in the image
ANTHROPIC_API_KEY -> secret manager, rotated
LLM_REQUEST_TIMEOUT -> explicit, e.g. 60s
PROMPT_VERSION -> pinned per feature, flag-controlled
# Infra must allow streaming: raise proxy/LB idle timeouts,
# disable response buffering on the streaming route.You cannot reproduce a bad answer from a stack trace. From day one, log every call's model, prompt version, token counts, latency, and a hash of the input, and trace requests end to end. When output quality drifts, this is the only way to find out what changed.
Deploy the boring parts the boring way, and spend your attention on the three that are genuinely different: a gateway to centralise model access, streaming end to end, and observability rich enough to debug outputs you cannot deterministically reproduce.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.