How prompt caching silently breaks in production

Prompt caching is the best-performing cost lever in LLM infrastructure, and it fails quietly when it fails. The failures do not raise an error, they raise the per-token input cost on a schedule you did not pick — a serialization quirk, a routing table that reshuffles keys, a failover to a different model family. Each of them looks like "the invoice is up." This post covers the three failure modes, with the paper trail that actually exists for them.

The fine print of a hit

The cache is a hash of a prefix, not a flag on your prompt. A hit requires:

• The exact same bytes, in the same order, as a recent request that wrote the entry.
• A prefix length above the provider's minimum — below that, every request is cold.
• Respect for the TTL: cache entries expire on the provider's clock, not yours.
• Often, the same account or key on the same path. Cache scope is a property of the endpoint, not of the model.

A hit bills at a fraction of input price (commonly ~10x cheaper on the cached portion). A miss that re-writes the entry often costs more than a normal input (a write premium). So a broken cache does not just lose you the discount — actively actively costs you money on every request.

Failure 1: serialization order changes the prefix

Providers hash the request as it arrives on the wire. A proxy or SDK that reorders fields — putting messages before system, or re-flowing a tool schema — changes the hash even though nothing a human would consider "the prompt" changed. Real instances:

• Bifrost issue #6406: on Vertex, prompt cache mostly misses because messages is serialized before system/tools, changing the prefix from what the upstream wrote.
• LiteLLM issue #35908: Vertex/Claude-Sonnet prompt cache does not reliably hit across multi-turn conversation.

Symmetric fix is a diff of the bytes on the wire for request N that you believe is a repeat of request N-1, up to the cached prefix length. If a diff is not empty, your cache is dead and there is no error to find it.

Failure 2: routing and failover (the one you can fix now)

This is the failure mode where your own routing table is the bug:

• Failover to a different family is a cold cache. If provider A goes down and you route to provider B's equivalent model, the cache does not follow your request. Worse within one provider: an API fallback from one model to another within the same family silently discards the prompt cache (claude-code issue #83272). And GPT-5.6 specifically: Codex cannot emit prompt_cache_breakpoint on it (issue #35300) — the cache is wired differently for that release.
• No sticky routing = no cache. If your endpoint round-robins across API keys or regions, each request may land on a different cache scope by design. Zoo-Code issue #1277 is the canonical report: routing without session affinity causes systematic misses.
• The economics of it: a failover cycle is a write premium on the missed prefix plus lost read discount on every request until the new scope warms up. Per failover task, that is commonly a 2–3x on task cost — not a rounding error. This matches the community field report that failover saved uptime while quietly tripling per-task cost (thread).

The flip side of the same coin is the best argument in favor of caching: a seven-location restaurant running Sonnet on every Instagram DM at a 97% cache hit rate — "97% cache hit is the only reason it's affordable." The hit rate is the business model.

Failure 3: upstream invalidation you did not trigger

Providers change cache behavior without a changelog event you will read:

• LiteLLM's July incident post tracked a Bedrock path invalidation for Claude Code prompt caches — the cache was evicted server-side, and the per-token cost spiked for everyone on that path simultaneously.
• Model version bumps (e.g. a minor revision of a "same" model) can change the hash scope. Your prefix bytes did not move; the upstream did.
• llm_engine #264: enabling Anthropic prompt caching on the Bedrock path is a configuration you do not have set until you set it, not a default you get.

The signal isinvoice-shaped: your request shape did not change, your per-token input cost rose, and the movement is correlated with a provider event, not with your release.

The arithmetic on getting caught

Illustrative numbers (real multipliers vary by provider — verify in the catalog): assume a 20k-token stable prefix (system + tools), input list of $X / Mtok, cache read at ~0.1x, cache write at ~1.25x.

• Healthy, cache-warm task: input ≈ 20k × 0.1X + a few fresh tokens ≈ ~0.2x of uncached.
• One failover wipe: input ≈ 20k × 1.25X + the new scope cold on the next traffic.
• Ratio: the wiped task costs ~6x the warm task in input alone. Output tokens do not change. This is the shape of a "our bill doubled" incident where nothing else in the stack changed, and it occurs once per failover event, not per user.

The check that catches all three

Every response carries the answer. Read it, per request:

def cache_health(usage):
    ptd = usage.prompt_tokens_details
    total = usage.prompt_tokens
    hit = (ptd.cached_tokens / total) if total else 0.0
    if hit < 0.8:
        alert(f"cache hit rate dropped: {hit:.0%}")

Aggressively, the same three numbers you already log:

• Hit rate per key and per task (target is high in steady state; a drop is an event, not a trend).
• $ per 1k input tokens, compared against your cached expectation.
• Failover count, so cache drops and route changes show up in the same timeline.

A cache hit-rate alert is a twenty-line job per provider you use. The alternative is discovering a routing mistake at the end of the month, on a line item, with an apologetic.

Per-model cache support and multipliers are in the model catalog; the per-request usage fields are documented in the API reference.