A rate card says 2.00 per million input tokens. It does not say how your text will be counted. For buyers of Anthropic models from the 4.7 generation forward, that omission matters: the newer tokenizers emit roughly 30 percent more tokens for the same input text. Unit price unchanged. Unit count up by a third, for the same prompt. Effective price per task: up by a third.
This has a name in consumer economics, and it is not a price increase: it is unit shrinkage. The package looks the same, the sticker is the same, but the number of units the package delivers quietly shrank - or, in this case, the number of units per serving quietly grew. The rate card did not raise your price. The meter did.
Why the tokenizer controls your bill
Tokens are counted by the model vendor's tokenizer, applied to your text. Tokenizers are trained per model family: vocabulary size, byte-pair-merge frequencies, handling of code, CJK text, and markdown all shift the token count between generations. A 30 percent count difference on identical input between two generations is a large number - large enough to reorder your entire cost ranking of tasks.
Two things make this a buyer problem rather than a lab curiosity:
- You compare across generations. Most of the "model A is cheaper than model B" math circulating around compares different tokenizers on the same text - or the same tokenizer on different text. Both are wrong. The honest unit is dollars per task, not dollars per million tokens.
- The drift is not obviously reversible. Fewer tokens per word is not automatically better for the model, and a provider training a new vocabulary is not optimizing for your bill. A vendor-optimal tokenizer is not a buyer-optimal tokenizer.
Indexing the drift
You do not need the exact tokenizer behavior to track it. You need an index:
- Pick 20-50 representative prompts - your actual top-50 tasks by volume, not toy examples.
- Tokenize each prompt with the tokenizer you are leaving, and record the counts against an index of 100.
- Tokenize the same text with the tokenizer you are moving to. The ratio to 100 is your drift factor.
- Multiply by both $ per million rates. The result is the real per-task cost comparison - the only number that should drive a migration decision.
| Step | Leaving (indexed to 100) | Arriving (measured) |
|---|---|---|
| Avg input tokens per task | 100 (index) | ~130 (reported for 4.7+ generation) |
| Rate ($ per 1M in) | 2.00 | 2.00 (Sonnet 5) |
| Effective input cost per task | 0.0002 | ~0.00026 |
(Numbers are shown in index form. Measure on your own corpus - the 30 percent is the reported average, and your mix of code, prose, and CJK will move it.)
The output side shifts too, and it shifts differently: the model decides when to stop, and a stronger model often writes more. The cheapest-looking $ per million hides a generation pair that talks more.
The effective token price you should track
Replace the single rate on the label with a rate per unit of your own work:
- Effective price per task = tokens emitted x the per-token rate, measured per task, per month.
- Keep the drift index in a version-controlled file, updated at every model change.
- When a provider ships a new model family, re-run the index before routing real volume at it - not after the first bill arrives.
A catalog that shows only $ per million is showing a price, not a cost. The cost is tokens times rate, and both factors moved without either appearing on the price shelf.
What to do
- Build a drift index from your real top-50 prompt corpus before your next model switch, and keep it as a regression artifact.
- Track cost per task in your billing view with $ per million as the input metric.
- Re-budget within a week whenever a provider changes tokenizers - announced or not. Check token counts on your standard prompts, since not every tokenizer change ships with a changelog.
- When reading any comparison, including the pricing comparison, check whether the compared models actually share a tokenizer family. The label does not tell you.