The standard inference node does two things in one place: prefill (read your prompt, build the KV cache) and decode (emit tokens against it). The problem is that the two phases have opposite hardware profiles - prefill is compute-heavy, decode is memory-heavy - so the unified node is half-wrong for each, and the fleet's effective utilization pays for it constantly. Disaggregated serving splits the work: a pool of compute-heavy nodes does the prefill, a pool of memory-heavy nodes does the decode, and the KV cache is moved between them over the interconnect for every request.
NVIDIA putting Dynamo 1.0 into general availability marked the shift from paper to production practice, and the ecosystem around it - Mooncake's distributed KV, the KV-aware schedulers in the llm-d line - is all the same story: the KV cache is now a moving asset, not a local variable.
The freight-leg accounting
The freight analogy is precise, not just decorative:
- The prefill pool is the loading dock. It receives the prompt and produces the KV - the "cargo" for the request - at a throughput that the rack's compute determines. Idle decode capacity is not the problem here; the problem is that a unified node spends half its hardware on prefill while it should be decoding.
- The decode pool is the delivery fleet. It emits tokens against the cargo, and its utilization is the whole value proposition: a memory-heavy pool, never interrupted by a prefill, decodes at a higher fraction of its theoretical throughput than a unified node can, because a unified node's prefill bursts strand its decode capacity.
- The freight leg is the KV transfer. Every request now has an in-flight MB-to-GB object moving between pools over the interconnect. It is fast - RDMA-class fabric - and it is not free: the bandwidth is booked, the transfer time is real, and it is a line item in the per-request cost that did not exist when the node was unified.
The utilization gain is the reason the shift is happening at all, and it is the same mechanism as a hub-and-spoke airline's: the specialized asset runs at a higher utilized rate than the general-purpose asset, because the general-purpose asset pays for its generality in idle cross-phase time. The freight leg is the price of that specialization - a freight charge, small per shipment, nonzero, and between two pools that are, increasingly, not in the same rack.
What it does to the buyer's observables
Three signals appear on the bill and in the traces that were not there before:
- TTFT and tokens-per-second can decouple. The prefill pool sets your time-to-first-token; the decode pool sets your tokens per second. A provider can be fast at one and slow at the other, and a unified-node mental model will conflate the two. If your TTFT improved but your decode rate did not (or vice versa), the disaggregation is the likely cause, and it is a config state, not a model state.
- The KV cache becomes a billable, movable asset. Prompt caching, which until now was a local memory optimization, is now an inter-pool problem: the cache that the prefill pool built is the very object the decode pool needs, and a router that can pin a request to the pool that holds its KV is buying a freight-leg elimination, not a speed boost. This is the architecture behind the prompt cache as inventory framing.
- The per-request cost has a transport term. It is small - the freight leg is a fraction of the token cost at current scales - but it is a new structural term, and it is the one that grows as context windows grow, because the cargo grows. Long-context requests are the heavy freight.
The watch items
- The interconnect is the bottleneck of the model, the way the hub is the bottleneck of the airline. If the fabric tightens, the disaggregation's utilization advantage compresses - and the freight leg stops being small. Watch for the shift from "utilization story" to "fabric story" in the provider's own architecture notes.
- The KV-aware schedulers (the llm-d work and its successors) are the next cost lever: routing a request to the pool that already holds its KV is a freight-leg elimination, and the providers who wire it at the router layer are the ones whose per-request cost keeps falling while the rate stays still - which is the same silent-deflation mechanism in the FP8 post, with a transport term.
- The context window is a fixed asset post frames what the KV reservation costs; this post is what the KV movement costs. Together, they are the two KV terms in the bill.
What to do
- Ask your provider whether prefill and decode are on unified nodes or disaggregated pools - the answer changes what your TTFT and your decode rate are measuring.
- If you run long-context, high-volume traffic, the KV transfer term is the one to ask about by name: how large is the cargo at your context length, and what is the per-request transfer cost at your scale.
- Optimize the cache reuse before paying for the re-prefill: a request that hits the KV cache on the prefill pool does not pay the prefill compute, and it has a shorter freight leg - that is the prompt caching mechanics with the transport dimension added.