Context Window Management in LLM Apps

Every LLM has a hard limit on how many tokens fit in a single call, and every provider charges by the token whether it's system prompt, retrieved context, or conversation history. Context window management is really two separate problems wearing one name: fitting inside the technical limit, and controlling what actually gets attended to once you're well inside it — a 200K-token window doesn't mean the model reasons equally well about token 190,000 as it does about token 500.

Treat the window as a budget, not a limit

The naive approach — keep appending messages and tool results until the API rejects the request — produces slow, expensive, and increasingly unreliable responses well before it produces an error. Set an explicit token budget per call, well under the hard limit, and allocate it deliberately: a fixed slice for system instructions, a slice for retrieved/grounding context, a slice for conversation history, and a reserved slice for the model's output. When a component would exceed its slice, trim or summarize it — don't let one runaway component silently eat another's share.

TOKEN_BUDGET = {
    "system_prompt": 800,
    "retrieved_context": 4000,
    "conversation_history": 3000,
    "reserved_for_output": 1500,
}
# total ceiling ~9300 tokens, well under a 128K window,
# leaving headroom and keeping latency/cost predictable

def build_prompt(system, retrieved_chunks, history):
    retrieved = truncate_to_budget(retrieved_chunks, TOKEN_BUDGET["retrieved_context"])
    trimmed_history = summarize_if_over(history, TOKEN_BUDGET["conversation_history"])
    return assemble(system, retrieved, trimmed_history)

Retrieval over stuffing

The instinct when a model gives a wrong answer is to add more context "just in case." That's usually backwards. Instead of including an entire customer record, order history, and product catalog on every call, retrieve only the chunks relevant to the current query via embeddings and vector search, and fetch the rest through tool calls only when the model actually asks for it. This keeps the average call cheap and fast while still giving the model a path to more detail when it genuinely needs it.

Summarizing conversation history instead of dropping it

For long-running agent sessions, don't just truncate old turns — periodically summarize them into a compact running state (open items, decisions made, key facts established) and replace the raw history with that summary. This preserves the information the agent needs to stay coherent across a long session while keeping token usage roughly flat instead of growing linearly with turn count. Re-summarize on a fixed cadence (every N turns) rather than only when you're about to hit the limit — reactive trimming under pressure tends to cut corners.

The "lost in the middle" effect

Published evaluations of long-context models consistently show attention isn't uniform across a large context: information placed at the very start or very end of the prompt gets weighted more heavily than information buried in the middle. Practically, that means the most decision-critical facts — the current user request, the specific record being acted on — belong near the end of the prompt, close to the instruction, not somewhere in the middle of a large block of retrieved documents.

Measuring context efficiency, not just cost

Track average tokens per call, but also track answer quality against context size — build an eval set where you know the correct answer requires specific facts, and confirm the agent finds them regardless of where those facts land in the assembled prompt. If accuracy drops as you add more retrieved context, that's a sign of dilution, not a sign you need a bigger model.

Wrapping up

Context window management is capacity planning, not a one-time setting. Budget tokens deliberately across system, retrieval, and history, prefer targeted retrieval over dumping everything in, and remember that where information sits in the prompt matters as much as whether it's there at all.