Cost Control for LLM Apps

LLM API costs scale with usage in a way traditional software costs don't — every additional user, every retry, every verbose system prompt shows up directly on next month's bill. Cost control isn't a one-time optimization pass; it's a set of defaults (caching, model tiering, output limits) plus ongoing monitoring, because the same feature that was cheap at 100 users a day can get expensive fast at 10,000.

Prompt caching is usually the biggest single lever

If your system prompt, tool definitions, or a large retrieved document is identical across many requests, both Anthropic and OpenAI support prompt caching that charges a fraction of the price for cached input tokens on repeat calls. An agent that re-sends the same 3,000-token system prompt and tool schema on every turn of a conversation is paying full price for static content on every single call — structuring the prompt so the stable parts come first and are cache-eligible, with only the variable part (the actual user turn) appended after, often cuts costs by more than caching a document once and querying it.

message = client.messages.create(
    model="claude-sonnet-4-5",
    system=[
        {
            "type": "text",
            "text": large_static_system_prompt,
            "cache_control": {"type": "ephemeral"},
        }
    ],
    messages=[{"role": "user", "content": current_turn_only}],
)
# static system prompt + tool definitions get cached;
# only the delta (the new user message) is billed at full rate

Not every call needs your most capable model

Route by task difficulty: a classification call (routing, tagging, simple extraction) usually does fine on a smaller, cheaper model, while multi-step reasoning over ambiguous data justifies the larger one. Building this as an explicit routing decision — not a single model for every call in the pipeline — is often a 3-5x cost difference on the calls that don't need the extra capability, with no measurable quality loss on those specific tasks.

Capping output length and retry budgets

Two silent cost leaks: uncapped max_tokens on calls that don't need long output, and unbounded retry loops on agent steps that fail. Set explicit, task-appropriate output caps — a classification call needs tens of tokens, not thousands — and put a hard ceiling on how many times an agent step can retry or loop before it escalates to a human instead of continuing to spend tokens on a task it can't complete.

Batching work that doesn't need a live response

For workloads that don't need an immediate answer — nightly categorization of a day's transactions, bulk document extraction — batch APIs (Anthropic's and OpenAI's both offer one) process requests asynchronously at roughly half the per-token cost of synchronous calls. Anything currently running as a loop of live API calls against a queue of records, where the result isn't needed within seconds, is a batch API candidate.

Tracking cost per outcome, not just total spend

A monthly API bill tells you what you spent, not whether it was worth it. Tag every call with the workflow it belongs to (case routing, contract extraction, forecast generation) and track cost per successful outcome for each — cost per case routed correctly, cost per contract extracted without a flagged error. That framing catches the case where total spend looks fine but one workflow's cost-per-success is quietly climbing because of a prompt regression or a model that degraded on that specific task.

Wrapping up

Cost control for LLM apps is mostly architecture, not negotiation with a vendor: cache what's static, route by task difficulty, cap what can run away, and batch what doesn't need to be live. Track cost per successful outcome per workflow so a regression shows up before the invoice does.