Token Optimisation for LLM Apps

Token cost in a production LLM application is a function of four decisions: how much context you send, which model you send it to, how much output you ask for, and how often you can avoid re-sending the same thing. None of these require exotic tooling — they're mostly design discipline applied before the first API call goes out, and they compound. A system that gets all four right can run at a fraction of the spend of one that gets none of them right, for the same user-facing quality.

Prompt caching: stop paying for the same context twice

Most production prompts have a large stable part — a system prompt, a tool schema, a set of few-shot examples, a document being repeatedly queried — and a small part that changes per request. Prompt caching lets the provider reuse the processed representation of that stable prefix across calls instead of reprocessing it every time. The practical effect is that a long system prompt or a reference document you query repeatedly stops being charged at full price on every request; you pay to write it into the cache once, then pay a much smaller amount to read it on each subsequent call within the cache's lifetime.

The catch is that caching only helps if the prefix is genuinely stable and appears in the same position on every call. If you interpolate a timestamp or a user ID into the middle of your system prompt, you invalidate the cache on every request and get none of the benefit. The fix is mechanical: put anything that changes — user input, retrieved context, session state — after the stable block, never before or inside it.

from anthropic import Anthropic

client = Anthropic()

# Stable block goes first and is marked cacheable.
# Per-request content goes after it, never interpolated inside.
response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=500,
    system=[
        {
            "type": "text",
            "text": SYSTEM_PROMPT,          # long, unchanging
            "cache_control": {"type": "ephemeral"},
        },
        {
            "type": "text",
            "text": TOOL_DOCS,              # also unchanging
            "cache_control": {"type": "ephemeral"},
        },
    ],
    messages=[{"role": "user", "content": user_query}],
)

Right-size the model to the task

Not every step in a pipeline needs the strongest model you have access to. Classification, extraction, routing, and simple reformatting are tasks a smaller, cheaper model handles reliably — the failure mode is usually ambiguity in the prompt, not a lack of reasoning capability. Reserve the larger model for the steps that actually require multi-step reasoning, synthesis across conflicting sources, or judgment calls with real consequences.

A common production shape is a cascade: a cheap model classifies or extracts first, and only ambiguous or high-stakes cases get escalated to the expensive model. This is more work to build than calling one model for everything, but it's also the single biggest lever most teams have, because model tier differences in cost are large while the task mix in a real pipeline is usually skewed toward the simple end.

Trim context: retrieve, don't stuff

The laziest way to give a model context is to paste the whole document, the whole conversation history, or the whole database row into the prompt. It works, and it is usually the most expensive possible way to solve the problem, because you're paying input-token rates on content the model never needed for that particular query. Retrieval — fetching only the passages relevant to the current request, via embeddings search, keyword search, or plain structured lookups — keeps the per-call context proportional to what the task actually requires.

The same applies to conversation history in multi-turn agents: summarising or truncating older turns instead of replaying the full transcript on every call keeps a long-running session from growing its per-message cost linearly for the whole conversation. A rolling summary of what happened more than N turns ago is usually enough context to keep the agent coherent, and it's far cheaper than re-sending every prior message verbatim.

Control output length, not just input

Input tokens get most of the attention because context windows are the visible constraint, but output tokens are billed too, and free-form prose is an expensive way to get a small amount of information out of a model. If the downstream consumer of a response is code — a database write, a UI field, another API call — ask for structured output (JSON with a fixed schema, or a short enumerated answer) rather than a paragraph the code then has to parse. This cuts output tokens directly and removes a parsing failure mode at the same time.

Setting an explicit max-tokens ceiling and a system-prompt instruction to be concise both help, but they're a backstop, not a substitute for asking a well-scoped question. A model asked "summarise this in one sentence" reliably produces a shorter, cheaper answer than one asked "tell me about this" and then truncated after the fact.

Batch and decouple non-interactive work

Anything that doesn't need a response within the same request-response cycle a user is waiting on — nightly categorisation jobs, bulk document summarisation, backfilling embeddings — should not be run through the same interactive, low-latency path as a chat interface. Providers that offer batch or asynchronous processing for non-interactive workloads typically price it lower than synchronous calls, in exchange for turnaround measured in minutes or hours instead of seconds. If a job runs on a schedule and nobody is watching a spinner, there's no reason to pay the premium for immediate responses.

This also decouples your cost from your peak concurrency. A synchronous pipeline that fires a burst of LLM calls whenever a batch job kicks off competes for the same rate limits and latency budget as your live user traffic; queuing that work and processing it asynchronously removes the contention entirely.

Wrapping up

Token spend is rarely one big inefficiency — it's the compounding effect of sending more context than needed, choosing a stronger model than the task requires, asking for more output than the task requires, and reprocessing the same stable content on every call. Fix those four in order of effort: caching and output control are usually a config change, model right-sizing needs an evaluation pass, and context trimming needs retrieval infrastructure if you don't already have it. None of it is exotic; it's the same discipline as trimming an over-fetching database query, applied to a different kind of API.