Open the 2026 vendor landscape and you will find five, sometimes seven, products calling themselves "the AI control plane" or "the LLM gateway," each drifting a little further from the others. It is not sloppiness and it is not paranoia: the word gateway has stretched to cover four distinct infrastructure jobs, and buyers keep mis-purchasing on the name. This is the taxonomy: the four layers, what each one does, and who actually needs which.
Why the labels exploded
Three things converged in 2025-2026, and they landed in different parts of the stack:
- Routing became universal. Every team running more than two or three model classes needs "which model answers this request" to be a decision with a config file. The router — policy on class, price, latency, or availability — matured into a standalone job.
- The unified wire matured. An OpenAI-compatible surface plus pass-through billing plus BYOK means one logical endpoint can front every provider, and platform vendors merged previously separate products into it — Cloudflare's own unification is the clearest example of a gateway absorbing what used to be three separate products.
- Observability and governance crowded in. An evals-first product shipped a router — LangSmith's LLM gateway is the visible case — while gateway-first products shipped policy consoles. The same drift shows up at the enterprise vendors consolidating "control plane" suites, in the open-source world where a Rust rewrite of a popular proxy put a fast layer-2 in the hands of teams that want to own it.
The practical consequence: vendors are re-labelling which layer they own. The buyer's job is to ignore the label and count the jobs.
The four layers
Layer 1 — model router. Decides which model answers the request: per task class, per price/latency target, per fallback chain. It is a config file with policy, no state and no billing. Good looks like: per-request policy (not per-team defaults), failover that preserves the model class rather than silently reaching for a different one, and logic you can review in a file.
Layer 2 — the gateway API surface. One OpenAI-compatible endpoint, one key, one metered bill, explicit failover and 429 semantics, BYOK or provider-billed pass-through. This is the layer that absorbs provider churn — deprecations, rate-limit shape changes, billing format changes — so your app does not. Build/buy reality: layer 1 is doable in-house; layer 2 is where the provider plumbing and billing infrastructure actually accumulate.
Layer 3 — governance / the control plane in its literal sense. Policy, data residency, approvals, key scoping, audit trail. The only layer that is enterprise-shaped: solo teams have no job to do here, and adding it early is buying a dashboard with no tenants.
Layer 4 — observability & evals. What actually happened: logs, traces, cost per request and per key and per task, and evals of output quality joinable on the same row as the cost. "Dollars per accepted answer" needs both columns on one row — which is why the evals tools and the observability tools keep merging.
Who needs what
- Solo / small app: layers 1-2. A router config and one wire is a complete architecture; adding three or four before you have traffic is buying hypothetical governance.
- Product team: add layer 4 early. Evals without cost joined on the same row are anecdotes; cost without evals is a bill with no opinion.
- Enterprise / compliance: add layer 3. Residency, approvals, and the audit trail are where "control plane" stops being a label and becomes a job.
Buy vs build, per layer: 1 is DIY-able (it is a file), 2 usually pays to be bought (provider plumbing + billing infra), 4 is optional enough that open source covers most of it, and 3 is the only layer where vendors earn their enterprise pricing — and the only layer that should never live in a config file.
The vendor questions that separate the layers
For any candidate, in any order:
- What per-token price do you charge? The answer is either the provider list price, or the list price times a markup — and if there is a markup, on which line of the invoice does it appear?
- Failover: what triggers it, and to which class? A 429 that trips a hop is not the same event as a timeout, and the destination class of a failed request is a cost variable, not an implementation detail.
- Cache semantics: provider-native prompt caching passed through, or a vendor cache layer with its own pricing and invalidation? The two behave and bill differently.
- Data residency: where does the request content live — logged, evaluated, at rest — and can it be pinned?
- BYOK and exit cost: you can bring provider keys in — in what format does the ledger come out when you leave?
Questions one and five will kill more "control plane" demos than the middle three: a markup, no matter how small, and a ledger that cannot be exported, describe everything about what the platform is optimizing.
The router and the wire — layers 1-2 — are what a gateway product is; the model catalog is where they are visible as pricing and routing data, and the exact wire semantics, including failover and 429 behavior, are in the docs. Layers 3 and 4 are different hats; keep them out of the label.