The standard mental model of context pricing is a prompt: you send more text, you pay for more tokens. The accurate mental model is a reservation: every concurrent request with a long context holds a slice of the provider's HBM - its KV cache - for the whole life of the session, whether or not the session emits another token next. The provider is carrying that reservation as a committed asset, and the long-context price multipliers on the rate card are the provider's balance sheet, itemized, onto your invoice.
This is the accounting that most of the context window explainer material skips, because the explainer frames context as a feature and the billing frames it as an asset. The two frames disagree, and the disagreement is where the cost is.
The reservation, made concrete
The KV cache is the working memory of a request: the per-layer key and value blocks that decode appends to on every token. Its size scales with the context length, the model's layer count, and the batch factor - and it lives in HBM, the same memory the bandwidth post identifies as the binding constraint on the decode phase.
Two consequences are the actual point:
- The reservation is a concurrency-tax. A provider serving 1,000 concurrent requests each holding a 100K context is carrying the KV for 100M context-tokens, whether or not the sessions are active right now. The HBM is booked. The billing on that HBM is flat - it is a fixed commitment - while the tokens emitted against it are variable. The provider prices the reservation against the committed HBM, and the price you see is the provider's utilization of that commitment, or the lack of it.
- The multipliers are the reservation, itemized. OpenAI's 272K-class threshold, where input bills at 2x and output at 1.5x, is the cleanest public version of this: the tokens past the threshold are not more expensive per token, the class of the request is more expensive, because the request's reservation is a larger slice of the fixed asset, and the provider is charging the asset class, not the token.
The buyer's side of the balance sheet
The fixed-asset frame changes three decisions:
- The blended $ per million is the wrong number for long context. When your context distribution is bimodal (short interactive, long document), the single blended rate is a lie in both directions - it overstates the cost of the short traffic and understates the cost of the long. The correct number is per-class: the reservation price, for the long-class traffic, and the token price, for the short.
- A long-context request is a capacity-booking, not a big-prompt. It is the same class of decision as booking a whole aircraft rather than a seat: the provider commits an asset for the duration, and the price of the booking is the asset's carrying cost over the duration, at the provider's utilization. Which means the price is only correct if the provider's utilization is real - and that is the watch item below.
- The concurrency tax is the tax on the buyer's own batching. A buyer who fires 50 concurrent long-context sessions, each holding a 100K context, is paying the reservation 50 times, at a level of concurrency the model's design does not need. The tax is the provider's carrying cost, passed through, and it is the one KV cost the buyer controls directly.
The provider's side, and why it moves
The reservation is the provider's committed HBM against the session's life, and the provider's economics are the utilization of that commitment. Three forces push the reservation price in both directions:
- More concurrent long-context traffic raises the provider's committed HBM, and the price of the asset class with it - that is the repricing being tested in the 272K-class multiplier, and it is the reason the threshold exists.
- KV compression (the FP8 and the quantized-cache line) shrinks the bytes-per-context-token, and the reservation's HBM-share, and the price of the class - that is the silent deflation in the FP8 post, hitting the reservation term directly, not just the token term.
- Disaggregated serving moves the reservation between pools - the freight leg in the KV transfer post - and a reservation that can be pinned to the pool that holds it is a reservation that costs less to carry.
What to do
- Split your context distribution into classes and price the long-class at the reservation multiplier, not at the token rate - the blended number will misstate your cost, in the direction that hurts you.
- Watch your own concurrency of long-context sessions. It is the one KV cost that is a buyer's operating decision, and it is the tax the provider passes straight through.
- When a provider adds a context-threshold multiplier, read it as a balance-sheet itemization, not a policy: the threshold is where the provider's asset class steps, and the multiplier is the carrying cost of that step, at the provider's current utilization.
- Re-derive the class prices after every KV-compression cycle - the reservation's HBM-share is falling, and the class price is the number that will follow it.