The KV cache - the working memory of every request - is now routinely stored at 8-bit precision instead of 16-bit, across the major serving stacks, and the newest quantization work pushes the compression to 3.76x on top of the halving. The provider's cost per token is falling, in a specific and measurable line of the cost stack, and the rate card is not moving. This is not a negotiation failure. It is the standard mechanism, and it has a name: silent deflation, the list-price-holds-while-COGS-falls pattern that the disk market ran for thirty years and the DRAM market runs every year.
What the quantization actually compresses
The KV cache is the per-layer key and value blocks, and its bytes-per-context-token scale with the model's precision storage. Halving the precision halves the bytes: 16 bits to 8, in the FP8 line that has been validated in production serving stacks, means the same context length occupies half the HBM. The 3.76x figure from the quantization work is the step past the precision-halving - the compression of the values themselves beyond the format change.
Two things fall with the bytes, and they are the different cost terms:
- The reservation term. The KV's HBM-share shrinks, and the concurrency tax in the context balance-sheet post falls with it, at the same context length. A provider carrying 100M context-tokens of KV at FP8 is carrying half the HBM the same load carried at FP16, and the carrying cost of the class is the number that reprices.
- The bandwidth term. Decode reads the KV on every token, and a smaller KV is a smaller read - the bytes-per-token term in the bandwidth arithmetic falls, and the tokens-per-second at the same hardware rises, or the same tokens-per-second at less memory. The speed and the cost are the same lever, pulled in the buyer's direction, invisibly.
Why the list price does not move
The silent-deflation pattern has a standard shape, and the LLM market is running it on schedule:
- The COGS falls in a specific line - here, the KV bytes - while the list price holds. The margin widens, and the widening is the provider's buffer against the repricing that will come.
- The repricing comes at the rack replacement cycle, not the model cycle. The provider reprice when the fleet turns over, because the deflation is a hardware-generation effect - the new racks are the ones that carry the FP8 profile - and the price follows the fleet, not the announcement. The disk market's per-bit curve is the template: the price holds, the generation moves, the per-bit falls at the refresh.
- The buyer's repricing entitlement is the delta. The portion of your current rate that is KV-COGS is the portion the provider does not need, and the delta between your current rate and the post-deflation COGS is the repricing you are owed, at the next fleet cycle. It is the same math as the floor in the power-contract post, with the KV term falling inside it.
The quality shift it hides
The one line of the pattern the list-price frame does not show: the compression is a lossy operation, and "validated in production" is the standard for the average case, not the edge. A quantized KV changes the output distribution in ways that are invisible at the benchmark and visible at the long tail - the rare reference, the exact citation, the numeric precision that the quantization's error term touches. The shift is small enough that no single eval will flag it and large enough that a serious buyer will feel it on the tasks that need exactness.
The buyer's defense is the same as for the tokenizer drift: a controlled A/B, token-counted, on your actual tasks, across the precision change. The list price will not tell you the quantization changed; your regression suite will.
What to do
- Track the KV precision your provider is serving, by model and by tier - the FP8 vs. FP16 distinction is a cost term and a quality term, and it is not on the rate card.
- When a provider adds or moves a context multiplier or a speed claim, check whether the explanation is the fleet cycle (compression) or the model cycle (a new generation) - the repricing entitlement is different for each, and the fleet-cycle repricing is the one that is owed.
- Run the controlled A/B across the precision change on your exactness-sensitive tasks, before you let the quantization propagate into production.
- Build the KV deflation into the cost model as a scheduled event, at the fleet cycle, not as an ad-hoc discount - the silent deflation is a feature of the stack, and the feature is recurring.