The hidden costs of LLM failover: cache invalidation, context replay, and tier drift

Failover is measured in uptime, but it is paid for in tokens. The first failure costs little — a request fails, a retry lands on a working provider — but the hop itself changes three cost variables of every subsequent request: how much context must be re-sent, whether the cache still applies, and which model class actually serves the work. Those are the hidden costs, and they compound: each request after the hop pays the worse number.

This post is what a hop does to each variable, what it looks like in a usage log, and the design rules that keep failover expensive-but-bounded.

Cost 1: context replay

Failing over mid-conversation means the next request to the destination carries the full state: system, tool definitions, history, in-flight tool results. The hop itself may cost nothing; the replay is not free — it is input tokens at full rate, on every step that follows while the session runs on the new provider.

Part of the mitigation is design: if your conversation state lives in your own store and is rebuilt per request, replay size is deterministic and independent of which provider you land on — a stateless handoff. The other part is where the handoff happens: failover at a task boundary costs less than failover mid-task. Failing between step 17 and 18 means the pre-hop requests already paid their full contexts once, and the hop adds one duplicate replay of the largest context the run has ever held. Failing at the task boundary means the duplicate is small.

Cost 2: cache invalidation

A prompt cache built on a stable prefix does not port across the hop. The destination provider — even one serving an equivalent model class — has a cold cache, so a run that read 90% of its prefix as cached foregoes its discount and pays the full input path again, including the cache write premium, since the destination must build its own cache from scratch.

The subtlety is the write asymmetry: writing a cache costs more than plain input, and it only amortizes if the prefix is re-read. A session that flaps — provider A, B, A, B, each stop starting cold — pays the write premium repeatedly for nothing. That is why the failover decision has to key on sustained failure, not on a single 429: one transient burst is a backoff, and flapping on it is a bill.

Cost 3: tier drift

The destination provider's equivalent model is not usually the same model class. A hop from frontier X to a "comparable" Y often means: the same advertised capability envelope, a different output-heavy token rate, a different cache discount, and in some pairings a quietly higher effective class for the workload. A task that was running mid-tier can end up paying frontier-class output rates for the rest of its life — with no error, no warning, just a thicker bill.

Log the drift as an event. When a request's model class changes across a hop, the ledger row should record it before the token totals: source class, destination class, the rate class that actually applied. Then the anomaly query becomes "spend rose and the model class drifted" instead of just "spend rose" — a different, actionable signal.

What it looks like in the log

A usage ledger after a failover event, one row per request, in the shape the cost tracking post describes:

ts         key        model     in/out/cached   class   note
09:14:02   prd-chat  aa-mid    3120/ 410/2900  mid     warm cache on provider A
09:14:31   prd-chat  bb-mid    3610/ 120/    0  mid     hop A->B, prefix cold, written
09:14:58   prd-chat  bb-mid    3409/ 140/3300  mid     destination cache warm again

The mid-tier class survived the hop — the model column never moved. The "cached" column did: from 2,900 to 0, and back again once the destination cache warmed. Uptime held, and the bill for that session is heavier by exactly one full-rate prefix plus a write premium, plus whatever the in-flight window adds.

Design rules that keep it bounded

1. Fail within class first. The fallback order should preserve model class — mid to mid, top to top — before it ever allows an escalation or a silent downgrade.
2. Hand off at the boundary when you can. A task-boundary failover is strictly cheaper than a mid-stream one, for the reason in the replay section.
3. Log the hop before the tokens. Class drift recorded at the endpoint, so the postmortem shows what the hop cost, not just that it happened.
4. Alert on sustained drift, not a single hop. One 429 and one hop is normal operation. N consecutive failures and a class change is a preference change, and preference changes should be a decision, not a reflex.
5. Express the SLO in cost terms. Not "99.9% of requests succeed" but "cost per completed task, failover headroom included, within X% of the no-failover baseline." That is the number the rest of the rules are buying.

A well-built failover is one where the bill's worst case is known: replay bounded by where the handoff happens, the write premium paid at most once per destination, tier drift logged and bounded by the fallback order. How a metered endpoint exposes each of those in your ledger is in the docs; comparing the source and destination classes before you configure the order is what the model catalog is for.