Caching Strategies for LLM Apps

Every LLM call is a network round trip you pay for by the token, so caching is the highest-leverage optimisation in most apps. The trap is that a naive cache serves the wrong answer confidently. Three different things get called "caching" here, they solve different problems, and you usually want all three.

Exact-match response caching

The simplest layer: if the exact same request comes in again, return the stored response instead of calling the model. The whole game is the cache key. It must include everything that changes the output — the full prompt, the model id, and the sampling parameters — because a request at temperature 0.9 is not the same request as one at 0.

import hashlib, json, redis

r = redis.Redis()

def cache_key(messages, model, params):
    payload = json.dumps({"m": messages, "model": model, "p": params}, sort_keys=True)
    return "llm:" + hashlib.sha256(payload.encode()).hexdigest()

def complete(messages, model, params, ttl=3600):
    key = cache_key(messages, model, params)
    hit = r.get(key)
    if hit:
        return json.loads(hit)
    resp = call_model(messages, model, params)   # your provider call
    r.setex(key, ttl, json.dumps(resp))
    return resp

Only cache deterministic-enough calls. At temperature 0 exact-match caching is safe and effective; at high temperature you are caching one sample of many, which is fine for cost but wrong if you wanted variety. Bound it with a TTL so a prompt or knowledge change eventually flushes, and include a prompt-version tag in the key so you can invalidate instantly on a deploy.

Provider prompt caching

Exact-match caching only helps on repeated identical requests. Provider-side prompt caching helps on the far more common case: a long, stable system prompt or document context reused across many different user questions. Anthropic and OpenAI both cache the prefix of your prompt so the repeated tokens are billed at a large discount and processed faster. Structure prompts so the stable part — system instructions, few-shot examples, retrieved documents — comes first and the volatile user turn comes last, then mark the stable boundary for caching. Reordering a prompt so the cacheable prefix is contiguous is often a bigger saving than any application-level cache.

Semantic (embedding) caching

If you want a hit when two questions mean the same thing but are worded differently, embed the query and look for a near-neighbour above a similarity threshold in a vector store. This is powerful for FAQ-style traffic and dangerous everywhere else: "what's my refund policy" and "what's your refund policy" should hit, but a threshold set too loose will serve one customer another customer's answer. Keep the threshold conservative, scope the cache per tenant, and never semantic-cache anything with user-specific data in the response.

Measure hit rate and the cost delta, not just latency. In practice a modest exact-match layer plus provider prompt caching on a long system prompt removes most of the bill for a support or retrieval app, and semantic caching is the optional third layer you add only once you have the eviction and per-tenant scoping right.