Back to blog

Local models vs API: the crossover math for 2026

What request volume does it take before dedicated hardware beats an API bill? A formula with explicit assumptions — and where caching, batch, and frontier class bend the curve.

Mappace Team · Engineering2026-08-185 min read
CostsArchitectureGuides

Local-versus-API shows up in developer circles roughly monthly, in the same sentence each time: "my 27B saved me hundreds of dollars in one evening." Both sides of the argument cite benchmarks. The crossover depends on no benchmark — it depends on your token volume and your mix. Here is the arithmetic, with every assumption made explicit.

The two costs, written out

The API side is linear in tokens:

api_cost(mo) = in_tok * pin + out_tok * pout
             - cache_discount * cached_tok
             - batch_discount * batchable_tok

where pin/pout are the list prices of the class of model you would otherwise run locally.

The local side is mostly fixed:

local_cost(mo) = capex / life_months + power + ops

Hardware is bought once; the marginal token after that costs you roughly electricity. Inference at the edge research (arXiv 2603.23640) is a good read if you want the formal treatment of when offloading to local compute actually helps.

The crossover is the monthly token volume at which the two lines cross. Everything below it, the API is cheaper in dollars; everything above, the added API tokens are wasted on hardware you already paid for.

A worked example (illustrative — substitute your numbers)

Assumptions stated out loud so you can swap them in:

  • A 24GB-class GPU on single-tenant cloud at ~$0.35/hr — roughly $255/mo including network.
  • Or a one-time PCIe card at ~$1,500 with home power at ~$60/mo, depreciated over 36 months: roughly $65/mo before ops.
  • Add ops: setup, model updates, monitoring — budget 10–20% of hardware cost/month. Realistically ~$75–95/mo home, $280+/mo cloud.
  • A mainstream 27B-class open model fits in 24GB; throughput on a healthy node is on the order of tens of output tokens per second.
  • API prices: pick the class of model you would actually use — a mid-tier inference model, not the frontier flagship.Check live per-token prices in the model catalog before you lock the number in — rates move quarter to quarter, and that movement is itself part of the math.

Do the division: (local fixed cost) ÷ (API $ per Mtok − local marginal cost) = crossover in Mtok/month. For the home-GPU case, that can land in the tens of millions of output tokens per month — plausible for a busy team's consolidated load, implausible for a hobby project. Volume and mix decide the crossover, not vendor loyalty — and the claim that "the GPU is free" only counts from the moment your fixed cost is already sunk into it.

This is not precision; it is a shape. Put in your real hardware cost, your real token mix, and the shape tells you whether you are on the flat side or the rising side of the crossover.

Where the curve bends (the reasons the bar moves)

  • Prompt caching. For workloads with heavy repeat prefixes — system prompts, RAG context, agent scaffolds — the effective API input price drops toward the cache-read rate (a fraction of list). This bends the API line down, moving the crossover to the right — which raises the local case in the scenario nobody models.
  • Batch rates. Many providers offer ~50% off for async jobs with a 24-hour SLA. If 30% of your volume is batchable (summaries, label work, index jobs), the effective per-token rate drops again.
  • The portfolio effect. A local machine is one model. The API is dozens: a long-context model, a reasoning tier, a small fast model, an embeddings model. The moment your architecture uses more than one class — which is most, by 2026 — "local vs API" is a false choice and the real question is which lanes go local.
  • Latency. Pre-filling a 20k-token context on a single GPU takes perceptible time. On the API's parallel fleet it does not. If your UX is interactive, the latency column in the bar graph is real money, not a footnote.
  • Risk profile. The API has a known unit price (which can jump — see the GPT-5.5 repricing this spring) and an availability tail. The GPU has a known fixed cost and an on-call tail. Some teams correctly prefer their outage over someone else's list price.

The honest answer

  • Why locally: single-model workload, stable high volume, privacy or data-contract requirements, or a flat spend floor with no risk of running an unattended loop.
  • Why the API: multi-model architecture, variable volume, flat spend floor, or simply not wanting to be a compute on-call.
  • What is actually happening: teams that are large enough that the crossover matters tend to hybridize anyway — a frontier path to the API, a bulk path to a local machine, two pools with different budgets. A pass-through-priced endpoint makes the recompute honest, because the lane you keep on the API does not carry a hidden markup, and the momentum deciding which lane stays on it is real arithmetic, not guesswork.

Compare the model classes you would run, at current list prices, in the model catalog — and run the formula above with your traffic data before you buy or cancel anything. That is the entire due-diligence for a crossover decision.