Speculative decoding is the mechanism behind a class of "fast tier" throughput claims, and it is worth understanding as an arithmetic, because the break-even is a number, not a claim. The mechanism: a small draft model proposes a stretch of K tokens, and the target model verifies the whole stretch in a single forward pass - one pass does the work of the K decodes, if the draft is right. If the draft is wrong at position i, the target emits the correct token for that position and the rest of the stretch is discarded.
The cost structure, per proposed stretch:
- The draft cost - the small model's forward pass over the stretch. Small, but nonzero, and paid whether or not the draft is accepted.
- The verify cost - the target's single pass over the stretch. Roughly the cost of one target decode, because the target is doing one forward pass, not K.
- The savings - the K-1 target decodes you did not do, if the stretch is accepted.
The break-even is the acceptance rate at which the draft cost plus the verify cost is less than the K target decodes the mechanism replaced. It is a function of K, of the draft's cost relative to the target's, and of the acceptance - and the acceptance is task-dependent, which is the whole story.
The freight-consolidation frame
The mechanism is freight consolidation: the draft is the one that loads the trailer, the target is the one that checks the load, and the acceptance rate is the trailer's fill. Consolidation pays when the trailer is full - the fixed cost of the run (the draft's pass, the target's check) is spread over K cargo tokens, and the cargo moves at the trailer's speed, not the pallet speed. It loses when the trailer is half-empty - the fixed cost is paid for K slots, K/2 slots move, and the empty slots are the draft's missed tokens, paid for and discarded.
The trailer analogy gives you the two failure modes directly:
- A draft that is always half-right is a fleet that runs half-empty trailers. The consolidation cost is paid, the consolidation gain is not realized, and the "fast tier" is slower than the raw target for that task class - and more expensive, because the draft's cost is now on the invoice.
- A draft that is almost always right is a fleet that runs full trailers. The fixed cost is spread thin, the speed is the trailer's speed, and the break-even is comfortably cleared.
Which fleet you are is the acceptance rate, and the acceptance rate is the task's predictability.
Why the acceptance rate is task-dependent
The draft is a smaller model guessing the target's next tokens, and the guess is good for exactly the portion of the target's output that is predictable - the portion where the correct next token is the high-probability one. That portion varies by task:
- Exact-output tasks (code with a known idiom, structured JSON, a citation that must be exact) have a high-acceptance region - the draft is guessing a constrained output, and the constraint is the draft's best case.
- High-entropy tasks (open-ended prose, a plan that is genuinely branching, a creative output) have a low-acceptance region - the draft is guessing a distribution with no dominant mode, and the constraint that makes the draft good is absent.
The break-even is not a property of the tier. It is a property of the task, measured against the tier's draft. A tier that clears the break-even on your code tasks can be below it on your prose tasks, on the same fleet, the same day.
What the buyer sees, and what to verify
Three observables, in order of value:
- The throughput claim, at your context length. The mechanism's gain shrinks as the context grows, because the draft's pass and the target's pass both cost more per token at depth. A "3x faster" claim at 4K context is not a 3x claim at 64K.
- The cost, including the draft. A tier that runs speculative decoding has the draft's cost in the rate, whether or not the invoice itemizes it. The break-even is the acceptance rate at which that cost is covered, and the rate is the one that does not move while the acceptance does.
- The quality, controlled. The accepted tokens are the target's tokens - the mechanism does not change the output distribution on the accepted path. But the rejected stretches change the path the target is on, when the target is steered by the draft's partial accept, and the shift is the kind the quality-shift frame covers: invisible at the benchmark, visible at the tail.
What to do
- For every "fast tier" you use, confirm whether it runs speculative decoding - the answer changes what the throughput claim is measuring and where the draft's cost sits.
- Measure your acceptance rate per task class, by class, on the tier you route at - it is the single number that tells you whether the mechanism is paying for itself on your workload, at your tier.
- Price the fast tier against the break-even, not against the list: the draft's cost is a real line, and the acceptance is the variable that determines whether the line is a gain or a tax, per the latency premium and the task's predictability.