Every model has a ceiling on how much it can process in one request. That ceiling — the context window — is the single most common source of silent failures in LLM applications, mostly because exceeding it rarely produces a helpful error.
What the context window holds
The context window is the maximum number of tokens a model can accept and keep in memory at once. It covers:
- Your system prompt and instructions.
- The full conversation history.
- Any documents, code, or retrieved passages you paste in.
- The tokens the model is about to generate as output.
Input and output are not separate buckets. Your generated reply competes with your prompt for the same window, which is why a huge input can leave no room for an answer.
Tokenizers vary by model
A context window is measured in tokens, and tokens are the product of each model's tokenizer — the code that splits text into chunks. Two models with identical numeric windows may hold different amounts of your text, because their tokenizers split words differently. That is why a "100k-token" window is only an approximate measure of how much prose it can hold.
What happens when you exceed it
The failure mode depends on the provider, and none of them are friendly:
- A hard error if your input alone surpasses the limit.
- Silent truncation of the beginning of your conversation — the model forgets the oldest context without telling you.
- The model output getting cut off mid-sentence because input filled the window.
Silent truncation is the dangerous one. Your app keeps running, but the model has dropped the instructions or context you assumed were still there. It answers confidently about information it no longer has.
Estimating tokens before you hit the limit
You can avoid the cliff by estimating your input size ahead of time, not reacting after truncation. Use the token count your gateway or SDK returns on the usage object of a normal request, and log it per conversation. When a conversation approaches a safe fraction of the window — say 80% — trigger compaction or retrieval rather than waiting for the model to silently forget. Headroom is cheaper than debugging a vanished instruction.
A worked example
Take an agent with a 32,000-token window running a long conversation. Early turns include a detailed system prompt and several documents. As the dialogue grows, it approaches the limit, and the oldest turns silently fall off. If your safety rules or product instructions lived in those early turns, the model has now "forgotten" them while you still believe it is enforcing them.
Staying inside the window
Three techniques cover nearly every real-world case:
- Truncate or summarize old turns — keep only the recent turns plus a running summary of older ones.
- Retrieval instead of paste — pull in only the relevant documents rather than everything.
- Auto-compaction — let the gateway or your framework fold old context into a summary when you approach the limit.
Each trades a little fidelity for the guarantee that the model sees some of the relevant context rather than none of it.
Whichever technique you choose, apply it before the window fills, not in response to truncation. Retrofitting context management onto an already-overflowing conversation is exactly the kind of last-minute fix that frustrates users.
Choosing a window
Pick a window that covers your worst-case input, not your average. Long agent loops and document Q&A need headroom; classification and extraction need almost none. Compare available context windows side by side in the model catalog, and read the docs to see how models map to a single endpoint.