The classic load balancer survives on one rule: any replica can serve any request. The rule is what makes scaling cheap - add the node, reweight, done - and it throws the information away, because inference carries a prefix artifact the classic balancer never had to price. Prefilling a prompt produces the KV cache (several megabytes) on the node that did the prefill, and the follow-up request with the same prefix - the standing system prompt, the conversation continuation, the template - either lands on that node and skips the prefill, or lands elsewhere and recomputes the megabytes and pays for them again. Disaggregation made the artifact infrastructure-level: Dynamo 1.0, general availability this year, split prefill and decode into separate pools, and the KV cache became a movable asset, with a transfer cost of its own itemized in the freight legs of KV transfer. Routing that accounts for cache location - KV-aware scheduling, affinity routing - is the load balancer that remembers the data. The two implementations to track are llm-d, with its KV-aware scheduling (whose published results, in the up-to-about-57x class on best-fit workloads, deserve direction, not fleet defaults) and Mooncake, the distributed KV store as the cache's substrate.
The precedent: the balancer that already remembered
A router that places by data location is not new; it rotated. Databases ran it in the nineties (consistent hashing, hot-spot replication, sticky reads), CDNs ran it by object, and the web ran it by session. Each learned the same five ways affinity fails, and the fleet is relearning them in KV form. The KV form adds one twist the predecessors did not have: the data (the cache) expires under memory pressure, because the HBM that holds the KV is the HBM that serves the batch, and evictions are continuous. The routing table's truth is time-decaying, and all five modes get worse with the decay.
The five failure modes, in the order fleets meet them
- The hot node. Affinity pins the traffic of a prefix to the nodes that hold its KV, and prefixes are Zipf: a handful of system prompts and templates carry most of the traffic, and the traffic pins to a small subset of the nodes, and the subset saturates while the long tail of the fleet idles. The imbalance is not a bug in the router; it is the benefit's residue. The mitigations are the known ones - replicate the KV across N nodes for a hot prefix, cap the affinity benefit below the load-delta where recompute beats waiting, spill to cold nodes explicitly - but the cap is a policy call, not an automatic, and the fleet that leaves it automatic buys its own imbalance.
- The stale affinity. The table says node seven holds the KV, and memory pressure evicted it ten seconds ago, and the request lands on seven, finds nothing, pays the recompute, and the table keeps lying until the next invalidation. The failure is quiet: no error, no loud log, just the P99 creeping and the recompute showing up in the cost line as a normal entry. The control is to treat the routing table as a cache - TTLs, invalidation signals, a staleness ratio as a measured metric - because a table that is 95 percent correct is a table paying a 5 percent recompute tax with the invoice un-named.
- The affinity to the failing node. The hot node starts to fail, the failover redistributes its load to the survivors, and the survivors lose their own cache in the same window (the import's memory pressure evicts the resident KVs), importing load from a node that was hot because of its cache. The cascade is the hand-off that evicts, and the loaded hand-off that inherits the eviction, and the failover's hidden cost is the hand-off's price, which the affinity makes into a load.
- The long-tail pinning. A request with a unique prefix has nothing to gain from affinity - the cache is cold by construction and no routing warms it - and pays for the affinity anyway: the bookkeeping, the pinning, the transfer attempts of the placement. A fleet with a high share of one-shot prompts (document batches, unique queries) carries an affinity that is pure cost on most of its traffic, and the measurement that finds it is per prefix class, not per fleet, because the fleet average hides the bleeding class.
- The overload the affinity refused. One application, one standing prompt, a burst: the affinity routes the whole burst to one node, where the stateless balancer would have spread it across the fleet, and the affinity converted a spread load into a point load, and the idle nodes are unreachable by policy. The per-client load cap above the affinity is the control, and the control exists only if the policy knows the client as a unit.
The cost view: when affinity pays
Per request, the decision is three outcomes, and the three have prices: the affinity hit (the compute avoided - input tokens times the input rate, times the fraction of the prefill skipped, plus the latency leg saved), the miss with transfer (the KV movement - a fraction of the recompute at rdma-class fabric, closing on the recompute over a public network), and the cold start (the full recompute). The router decides among the three, and the decision is right when the router knows the three prices and the three probabilities, and the probabilities are measurements, not configuration: the hit rate per prefix class, the avoid-recompute per hit, the load Gini across the nodes, the table's staleness ratio. The fleet that deploys affinity with the confidence and without the four measurements is buying the saving on the hot class and paying for the cold class, both invisibly, in the same line, at the same time.
The rule of thumb the practice keeps landing on: affinity pays on the prefix-heavy, high-frequency workloads - the chat with the standing long system prompt, the repeated few-shot battery, the templated pipeline - and loses on the one-shot batch work where every prompt is unique. A single always-affinity global policy is right for one class and a tax on the other, and the class is the routing dimension the routing strategies post did not carry, because the dimension is the data dimension, and the data dimension arrived with the KV.
What to do
- Instrument the four ratios, per prefix class: the affinity hit rate, the avoid-recompute per hit (in tokens and in dollars), the load Gini across nodes, and the routing table's staleness ratio. The fleet that can read the four can decide what the policy should be; the fleet that cannot is configuring by rumor.
- Set KV replicas for the hot prefix (two to three is the working range), and break affinity when a node's load crosses twice the fleet mean - a threshold, not a hope - because the hot node is the first failure mode and the one the policy exists to stop.
- Treat the routing table as a cache with TTLs and invalidations, and make the staleness ratio a first-class metric, because the stale entry's quiet recompute is the cost that reads as normal in the bill until the table is trustworthy.
- Separate the "affinity node died" from the "affinity node is busy" in the failover logic: the dead one loses the cache, the busy one does not, and the recovery paths for the two should differ, the same as the routing strategies post separates the model from the provider.