Blog
Practical writing on LLM APIs, model selection, and building production AI.
The 2026-07-28 MCP spec revision made the protocol core stateless: round trips are explicit, routing is by headers, and the listings carry cache semantics. That is an implementation detail on the page and the moment the agent protocol became measurable in the field. Metering has an order: round trips first, tool-list cache hits second, payloads third, per-tool latency last. Here is the meter the stateless core makes possible.
Routing across three providers is the standard redundancy answer, and the answer assumes the three are independent. The August 20-21 OpenAI cluster - the auth pileups, the 3-hour thinking-model degradation, the 55-minute Codex authentication window - is a case study in what the assumption hides: the shared substrate, and the shared dependencies, that make three providers one supply chain in the places that matter. Here is the read.
Stripe acquires OpenRouter for a reported ~$7.5B in August 2026. What it means for your keys and billing, and how the main alternatives actually compare.
A single OpenAI-compatible layer between your app and every model provider — one key, one API, transparent per-token pricing, and automatic failover.
Gemini 3.7 Flash's launch promo is 0.75 / 3.75 - and the same model, batched through a router, lands 87.5 percent below that. An aggregator undercut the vendor on the vendor's own promotional price. That is not a discount. That is a spread, and it has a margin structure worth understanding before you route around it.
DeepSeek V4-Pro bills the same token at two rates depending on the UTC hour - double during peak windows. That is electricity-grid demand pricing, applied to inference. Here is why it works there, and what it implies for the routing layer.
The August 2026 enforcement window turns data residency into a routing problem, not a legal curiosity. Practical checks: data classes, provider options, zero-retention, audit trails.
Repoint one base_url and keep your SDKs, templates, and tooling. What OpenAI compatibility means, and what actually breaks when you switch.
A new model state has arrived: not just available, not just expensive, but gated - visible, priced, and only partially buyable. OpenAI's Daybreak tier and June's 19-day government disablement put access, not price, on the model catalog. Here is the allocation economics behind it.
What request volume does it take before dedicated hardware beats an API bill? A formula with explicit assumptions — and where caching, batch, and frontier class bend the curve.
This year the frontier tier moved up - GPT-5.5 doubled, Fable 5 launched at 10.00 / 50.00 - while the mid tier held or fell. That inversion has a name from the consumer market. It tells you which SKUs to ration and which to flood.
Why moving from one model vendor to another is usually a rewrite today — and how a single OpenAI-compatible endpoint reduces it to one line.
OpenAI's Sol promo ends November 21, Gemini 3.7 Flash's promo ends December 31, and a Sonnet 5 price hike was announced then quietly withdrawn. The rate card is now a dated instrument. This is how to budget for it.
The EU's Digital Omnibus deferred the Annex III high-risk regime to December 2027 - 16 months of runway - while the GPAI and transparency obligations went live on August 2, 2026. The deferral is not a reprieve; it is a pacing signal. Here is the plan the runway implies, staged, with the work that starts now and the work that cannot wait for the runway.
Context window, reasoning depth, price per token, and latency — the four knobs that actually matter when picking a model, and how to read them at a glance.
GPT-5.6's tiered release, DeepSeek V4-P GA, GPT-5.5's May doubling, and Claude's shift to metered billing — how pricing moved this quarter, and how to track it monthly.
Cache reads are cheap; cache writes less so — and routing can invalidate both. The three ways prompt caching quietly fails across providers, and the check that catches each one.
What a reasoning model actually does under the hood, why the added tokens and latency are sometimes worth it, and how to know which tasks justify them.
Anthropic's mid-June billing change made coding-agent spend session-relative. What it changed, why retry loops exploit it, and where a metered endpoint with real caps fits in.
Input vs output tokens, why output is priced higher, how reasoning tokens inflate your bill, and how Mappace bills exactly the model's underlying rate.
Anatomy of a $6,000 LLM month: the three mechanics that spike an agent bill — unattended loops, context replay, invisible sub-calls — and the four controls that stop it.
What a context window holds, what silently goes wrong when you overflow it, and techniques like auto-compaction that keep long conversations running.
A 429 means three different things depending on the provider: a rate-limit burst, a spent cap, or a plan quota. How to decode the cause before you back off, fail over, or stop for good.
json_object, strict json_schema, and tool-schema constraints are three different guarantees — and they break on different providers. The round-trip test matrix and the patterns that survive switching.
Providers go down, rate-limit, and degrade. How automatic failover keeps requests flowing, the retry logic that matters, and how to design for it.
Planning calls, worker calls, context re-sent every step, and rework: the four line items behind an agent's bill, and the three levers that shrink each one.
Session windows, weekly budgets, per-model caps: what each clock measures, what an API key means by contrast, and the trap of routing subscription keys through tooling.
Prompt sizing, output control, caching, and model right-sizing — concrete levers that cut token spend before you reach for a cheaper model.
Failover is measured in uptime but paid in tokens. What a provider hop does to your context re-send, your cache state, and which model class actually serves the rest of the run.
How bring-your-own-key works, when keeping your own provider accounts makes sense, and how a gateway layers one endpoint over multiple keys.
Three archetypes — support chatbot, RAG assistant, coding agent — with worked monthly math, the variables that move each bar, and how to sanity-check the number against your own traffic.
Every model you have routed has an unannounced end date, and the migration it forces is a cost you already pay, in a line item nobody named. Model deprecation is an actuarial risk with a survival curve, a premium, and a hedge - and the premium is the re-validation cycle your team runs on a schedule no one wrote down. Here is the math that makes it visible.
Time-to-first-token, perceived latency, and the UI trade-offs of streaming — plus when a plain non-streaming request is the better call.
'Cheapest LLM' has no answer; 'cheapest for that task' does. A per-task price table — classify, summarize, chat, reason, code — plus the hidden dimensions that make the headline lie.
How to decide which model serves each request — round-robin, least-latency, failover, and rule-based routing — and when each one is the right fit.
Three queries are all 'cost visibility' actually means: who spent, on what, and when the shape went wrong. A minimal usage ledger answers all three — no platform required.
A 429 retried with backoff is not free error handling: each attempt re-bills the input, the backoff interval is a bet on when the residual capacity will clear, and the herd is correlated. Airlines solved this exact problem as yield management - a fare grid against the probability of boarding. Read the retry policy in that frame, it is the cheapest fare class on availability, the one nobody prices. Here is the math, and the grid the policy belongs to.
Five vendors now sell 'the AI control plane.' The label covers four distinct jobs: route, unify the wire, observe, govern. What each layer does, who needs it, and the vendor questions that separate them.
A concurrent request with a 200K context is not a big prompt. It is a reserved slice of HBM that the provider carries for the life of the session, whether or not it emits a token next. The long-context price multiplier is the provider's balance sheet, itemized, onto your invoice. Here is the accounting.
The $/M-token headline is one of six columns you actually pay: input, output, cache read, cache write, batch discount, context tier. Plus the markup line — and why the May price jump shows up in none of them.
The cheapest LLM endpoints keep pricing below the full cost the industry can verify - for months. That is not a bug and not a subsidy. It is loss-leader bidding against a cost floor that power, HBM, and amortized silicon pin at a detectable level. Here is the math the bidders are all running.
A subscription and an API key are two different contracts — one for human-paced work, one for machine-paced work. The break-even, where the math breaks for apps, and the two-pool pattern that keeps both true.
The majors run a batch tier at roughly half the standard rate, with a 24-hour completion window. That is not a durability discount. It is a forward contract: you sell the provider your timing, and the discount is the price of the immediacy you gave up. The spread tells you the provider's cost of real-time. Here is the read.
The prompt cache charges you a write fee (1.25x to 2x) to hold a copy, sells reads at 0.1x while the copy is fresh, and expires the copy on a TTL. That is inventory: restocking fee, shelf life, write-off. The frame changes when the cache pays and when it quietly costs you money.
On June 12, 2026, US export controls suspended Claude Fable 5 and Claude Mythos 5 for all users, and they stayed off for 19 days. It was the first time a shipped model was disabled by a government directive, mid-flight. It was also the first time a model contract had to answer a question it was not written for. Here is the question, and the clauses.
The 99.9 percent is the marketing. The service credit - the percentage of fees the provider will actually pay if the service fails - is the actuarial number: the provider's own price for the failure, calibrated to the provider's own downtime data. Read the fine print and the SLA stops being a promise and starts being a probability. Here is the read.
The same open-weight model costs 6x more at 750 tokens per second than at 150 - and a different class crosses 2,700 at a 280ms time-to-first-token. Latency is no longer a feature you get; it is a priced SKU with a measurable premium. Here is when paying it is a good trade and when it is a tax.
Llama 4 70B sells for about 0.65 per million tokens at 100 tokens per second on an H100 and about 4.20 at 750 tps on an ASIC endpoint - same weights, six price points, a 6x spread and a 7x speed spread. That is not a market with prices. That is a refining market with cracks. Here is how to buy it.
CVE-2026-2256 - the ModelScope agent framework's injection-to-RCE path, through a regex blocklist the payload walked around - is the confirmation the OWASP report was predicting: prompt injection is not a prompt problem, it is a perimeter problem. The agent has tool rights, and the injected text that reaches the tool is code execution. Here is the math the incident response frame puts on it.
OpenAI, AWS, and Google all sell region-pinned inference at a roughly 10 percent premium over the same model, same weights. Sovereignty became a priced SKU. Here is the tariff math behind the premium, and how to route only the traffic that must pay it.
Every LLM fleet is starting to route by which node already holds the prompt's KV cache. The saving (avoids recompute) is real, and the failure taxonomy is not new: it is data-affinity scheduling, the one the databases ran in the nineties, and the hot node is the first failure that bites. Here are the five failure modes, in the order fleets meet them, and the measurements that separate the win from the trap.
A small draft model proposes a stretch of tokens; the big model checks them in one pass. When the draft is right, you get free speed; when it is wrong, you paid for the stretch and got the pass anyway. The break-even is not a vibe - it is the acceptance rate at which the draft's cost is covered, and it is different per task. Here is the arithmetic.
The KV cache is now routinely stored at 8 bits instead of 16, and the newest quantization work claims 3.76x compression on top of that. The cost per token is falling while the price per token does not move - the classic silent deflation. Here is what it means for the repricing you are owed, and the quality shift it hides.
OpenAI's capacity lease with Cerebras - ~750 megawatts, roughly $10B, reported at about 0.47 per megatoken of reserved throughput - is the first time the input cost of inference was written into a signed contract. It also made inference capacity a listed asset. Here is the arithmetic underneath.
A chat completion bills at a fixed weight. An agent task bills a distribution, and the measurement that shows it - roughly 1000x the token spend of the comparable chat, with a tail that runs 10x past the median - is the measurement the budgeting frame needs. Value at risk, the airline's overbooking math, is the frame. Here it is, applied to the agent bill.
Strip a token down to its cost components and two things survive forever: the electricity and the amortized rack. Both are priced on contracts, both are public, and their sum is the floor below no rate can go for long. Here is the math, with every assumption labeled.
Anthropic's 4.7-generation tokenizer emits roughly 30 percent more tokens for the same input text, and the $ per million rate did not move. The sticker price stayed. Your effective price per task went up. Here is how to measure the drift and re-baseline.
Input and output tokens bill at different rates, and the meter only ever runs one way: it accumulates, never refunds. The 0.60-in / 2.40-out asymmetry is a design fact, not a coincidence, and your ledger has to be built around a meter that has no negative. Here is the accounting it implies.
Everything the vendor spec sheet sells is FLOPS. The thing that sets your tokens-per-second - and, downstream, your $ per token - is high-bandwidth memory. Decode is a memory-bound phase, and the fast tiers are priced by the bandwidth they bolted on, not the compute they added.
NVIDIA's Dynamo 1.0 general-availability marked the shift: prefill and decode, split onto different node pools, with your KV cache shuttled between them. It improves utilization the way a hub-and-spoke airline improves aircraft hours - and it adds a packet-transfer line item to the price of a token that did not exist before.
