LLM costs scale with tokens, and tokens are mostly under your control. Before you switch to a cheaper model — which can silently hurt quality — pull the levers that reduce tokens on the models you already trust.
1. Shrink the input side
Input is cheap per token, but it is usually where the volume lives. Fixing it has the largest impact on a per-request basis:
- Cut verbose system prompts down to only what the model needs.
- Send recent conversation turns plus a summary instead of the full history.
- Retrieve only relevant documents rather than pasting entire files.
- Deduplicate repeated context that does not change between requests.
Trimming input also improves quality, because a shorter prompt is less likely to be ignored or lost to truncation.
2. Control the output side
Output tokens cost several times more than input, so output length is the real driver of spend:
- Set an explicit
max_tokenson every call instead of letting the model run on. - Ask for the shape you want — concise, bulleted, or JSON — and the model will respect it.
- Avoid chain-of-thought prompting on tasks that do not need it; reasoning multiplies output cost.
A model that writes 300 tokens instead of 2,000 is the single biggest cost win available, with zero quality loss for most tasks.
3. Use caching
Many requests share context — a fixed system prompt, tool definitions, or the same document. Provider-side caching can reuse that input across requests rather than re-billing it. Structure your prompts so stable text sits at the start and volatile text at the end, where caches are most effective. The savings are largest when you have many near-identical requests hitting the same prefix.
4. Right-size the model
Not every request needs your most capable model:
- Route classification, extraction, and formatting to a fast, cheap model.
- Reserve reasoning models for the small slice of tasks that measurably need them.
- Compare models on your token mix, not on headline benchmarks.
Because every model sits behind one endpoint, right-sizing is a per-request choice, not a new integration.
5. Find the big spenders first
Not all requests cost the same, so optimization should start where the money is. Pull your request logs and sort by total cost. Typically a handful of call patterns — one long-context agent, one verbose summarization job, one fan-out classification — account for the majority of spend. Fixing those three beats shaving a few tokens off every request.
6. Batch what can wait
Real-time is expensive because you pay for latency you do not use. Classification, extraction, and summarization over a backlog of documents do not need sub-second responses, so run them as a batch — feed many inputs through the same prompt and let the provider chew through them without a human waiting on each one. Batching does not cut the token count, but it drops the operational overhead of thousands of tiny round-trips and lets you use slower, cheaper capacity off-peak.
7. Measure before optimizing
You cannot cut what you do not see. Track per-request input/output tokens and cost, then target the requests that dominate your bill. Re-run the same query after each change so you can confirm the savings are real and not offset by a quality drop you did not notice.
Cost is a moving target, so treat these levers as recurring practice rather than a one-time cleanup. Revisit your prompt sizing and model routing every time you ship a feature, because each new feature quietly changes your token mix.
See exactly where tokens go in your usage analytics, and compare per-token prices in the model catalog before you downgrade a model.