Read an inference vendor's spec sheet and it sells you one number: FLOPS, or its cousin, "performance per watt." That number is mostly wrong for the half of your bill that matters. The decode phase - the phase that emits your tokens one at a time - is not compute-bound. It is memory-bandwidth-bound, and the arithmetic is simple enough to fit on a line:
tokens per second, at decode, is approximately the memory bandwidth divided by the bytes of active model state that must be read for each token.
Every generated token requires re-reading the weights (or the active slice of them) and appending to the KV cache. Compute the addition and you have done nothing; move the bytes and you have a token. The FLOPS sit idle for most of the decode cycle. This is why an H100-class config emits 100-150 tokens per second on a 70B-class model, and an ASIC config - a chip design whose defining feature is memory bandwidth per dollar, not FLOPS per dollar - emits 600-750 on the same class of model, with one config putting an open-weights 120B at roughly 2,700 tokens per second.
The memory-market consequence
Stack that fact on top of what HBM is in the supply chain, and the cost structure of inference resolves into two markets:
- The FLOPS market (logic silicon). Competitive, scaling down in price per operation every year, and mostly irrelevant to your decode throughput.
- The HBM market (stacked high-bandwidth memory). The binding constraint, the vertical most often out of silicon-class supply, and the one whose price per bit and whose bandwidth per dollar determine your tokens per second more than anything else on the spec sheet.
The price of a token at the marginal provider is, to first order, a memory price. That is the full claim: the $ per million you pay for output tokens at the fast tier is set by the bandwidth the provider booked, and the bandwidth is set by the HBM they stacked. The FLOPS are the marketing; the HBM is the metering.
Why MoE fits the frame
The mid-tier price collapses of 2025-26 are largely MoE economics, and they fit the same structure: Mixture-of-Experts models activate only a fraction of their total parameters for each token, which shrinks the bytes-per-token term in the decode ratio. The same HBM bandwidth then yields more tokens per second, because there is less to read per token. The "cheap" in a cheap MoE endpoint is bandwidth-per-active-byte - not a magic price cut, but the same arithmetic with a smaller numerator.
What a buyer should check
Three questions that cut through spec-sheet marketing:
- What is the HBM bandwidth per accelerator, in your config, not in the brochure? The config number - the bytes-per-second the deploy can actually pull - is the only number in the decode ratio.
- How many active bytes per token? Total parameters overstate it; active parameters (the expert slice, for MoE; the layer's kv-blocks, always) is the honest denominator.
- Is the fast tier buying bandwidth, or FLOPS? A 5x throughput jump with the same FLOPS class is a memory story - which is also a story about the part with the tightest supply, and therefore the part most exposed to a supply-driven repricing, per the below-cost pricing analysis of what sits below the list.
The speed premium in the latency premium post is the consumer's name for this; the HBM market is the supplier's name for it. Same spread, two ledgers.
What to do
- For every tier you route at, record the config's bandwidth per accelerator and the model's active bytes per token - the ratio is your expected tokens per second, and the delta from the vendor's claim is your signal for how the config is sized.
- Treat a bandwidth-driven repricing as a watch item: HBM supply tightens and the fast tier is the first place the spread moves, because the fast tier is where the memory lives.
- Re-derive the ratio after every model generation - a new model with different active-width changes the denominator even on the same hardware.
- When a fast tier claims a throughput figure, ask for the model and context length it was measured at, because the decode ratio has two variables and vendors quote the favorable one in comparisons.