Every LLM rate card is two numbers, not one: an input rate and an output rate, usually a 4x spread (0.60 in / 2.40 out is a standard mid-tier pair). The spread is not a pricing accident. It is the cost structure of the two phases, priced honestly: input is prefill, a compute-bound phase; output is decode, the memory-bandwidth-bound phase in the HBM arithmetic. The decode is the expensive half of the token, and the rate card says so in the spread.
The accounting fact that follows, and the one most billing setups miss: the meter is directional. It accumulates input cost against the input rate, accumulates output cost against the output rate, and never debits. There is no negative token, no credit that runs the meter backward, no refund line. The bill is a sum of two one-way accumulators, and the design consequences for a ledger are specific.
Why the meter has no negative
The one-way property is not a billing-policy choice. It is the shape of the work:
- A token is emitted, or it is not. There is no "un-emitted" output token that refunds input cost. The prefill happens once, against the prompt, and the cost is sunk the moment the request fires.
- The rates are per-direction, and a direction's cost is the cost of that phase's hardware time. The decode that bills at 2.40 is the memory-bandwidth time; the prefill that bills at 0.60 is the compute time. The meter records the hardware time, per direction, and the hardware time is a one-way quantity: the phase ran, the hardware drew, the meter advanced.
- The 4x spread is the ratio of the two phases' hardware costs, at the provider's fleet. It is not a markup on the direction. It is the meter reading the two circuits separately, and the bill is the sum.
The design consequence: a ledger built for a one-way meter is a sum of per-direction accumulators, with no offsetting between directions, and the "effective rate" is a derived number, not a booking.
The derived number that misleads
The standard mistake is to compute an "effective rate" as the weighted average of the two rates, by token volume:
effective = (in_tokens x in_rate + out_tokens x out_rate) / (in_tokens + out_tokens)
The number is a real number, and it is the wrong number for the two decisions that matter:
- It hides the mix. Two workloads with the same in/out ratio but different volume have the same effective rate and different bills. The effective rate is a ratio; the bill is a product, and the product is the one that moves when the volume moves, at the same ratio.
- It hides the direction. A workload that shifts its mix (more output per input - a more verbose model, a longer answer) moves the effective rate without moving either rate. The effective rate moves, and the reader of the effective rate sees "the price changed" when no price changed, only the mix.
The correct booking is per-direction: the input accumulator at the input rate, the output accumulator at the output rate, the bill as the sum, and the effective rate as a derived view, labeled as a derived view, with the mix (the in/out ratio) next to it, so the reader sees which of the two moved.
The multi-provider complication
The one-way meter gets a second property when you route across providers: each provider's meter is independent, and the meters do not net. Provider A's overage does not offset provider B's underage; the ledger is a sum of per-provider, per-direction accumulators, and the "effective rate" across providers is a weighted average of independent meters, which is a portfolio number, not a booking.
The two consequences are the ones the gateway ledger has to handle:
- The per-provider meter is the unit of reconciliation. A cross-provider billing view that nets one provider's cost against another's is a portfolio view dressed as a booking, and it hides the per-provider drift that is the actual signal - the provider whose meter is running hot on your traffic is the one to watch, and the net hides it.
- The direction assignment can differ by provider. A task you route as input-heavy to one provider and output-heavy to another is two different bookings, at two different effective rates, for the same logical task. The mix is per-provider, per-direction, and the ledger has to carry that granularity or the reconciliation will not close.
What to do
- Book the ledger per-direction, per-provider: the input accumulator and the output accumulator, separately, and the bill as the sum. The effective rate is a derived display, with the in/out mix next to it.
- Watch the mix, not the effective rate, for drift. A mix shift is a workload change, not a price change, and the two have different responses.
- Reconcile per-provider, per-direction, at the meter level - the cross-provider net is a portfolio view, and the portfolio view is where the per-provider hot meter hides.
- When a rate card changes one direction (an output-rate reprice is the common one), the per-direction booking shows the exact slice of the bill that moved, and the effective-rate view shows a muddy average of it. The booking is the audit; the average is the dashboard.