Once you have more than one model behind a single endpoint, you need a rule for which one answers. That decision — routing — is where most of the cost and reliability of an LLM stack is actually determined.
How model routing works in practice
A router sits between your request and the set of candidate models. It applies a policy to pick a backend, sends the request, and returns the response. The policy can be as simple as a default model name or as complex as a probabilistic load balancer.
The main strategies:
Round-robin
Each request goes to the next model in a fixed rotation.
- Good for: spreading load evenly across providers you want to keep warm.
- Weakness: ignores latency and cost, so a slow or expensive model gets the same traffic as a good one.
Use round-robin when your candidate models are roughly equivalent and you mainly want to avoid hot-spotting one provider.
Least-latency
The router tracks recent response times and prefers the model that is currently fastest.
- Good for: interactive traffic where time-to-first-token dominates the experience.
- Weakness: needs live latency telemetry and can thrash between providers during noisy periods.
Use least-latency for user-facing chat where a few hundred milliseconds is the difference between "snappy" and "slow."
Failover
Requests go to a primary model and switch to a backup only on error or timeout.
- Good for: availability-first workloads, where the priority is "never drop a request."
- Weakness: does not balance load or optimize cost — it only covers failure.
Failover is a floor, not an optimization. Every production app should have it as a baseline, layered under whichever other strategy you prefer.
Rule-based routing
The router picks a model from request attributes — task type, cost ceiling, context length, or a model hint — using explicit rules.
- Good for: fine-grained control over cost and quality per task.
- Weakness: you maintain the rules, and a bad rule is a bad default.
Rule-based routing is where right-sizing lives: send classification to a cheap model and hard reasoning to an expensive one.
Weighted routing
A probabilistic cousin of round-robin: each model gets a fraction of traffic, say 70/30 across a primary and a challenger.
- Good for: gradual rollouts and tests — you migrate traffic to a new model in steps.
- Weakness: like round-robin, it ignores live conditions unless you layer on more logic.
Use weighted routing when you want to compare a new model against the old one in production without an all-or-nothing cutover.
When to graduate past static rules
Static policies get you far, but there is a point where you want routing that reacts to live conditions instead of a frozen config. The signal to upgrade is when you observe the same recurring trade-off a human keeps tuning by hand — a model whose latency spikes weekly, or a cost that oscillates with traffic. At that point, layer a dynamic policy over your static defaults, and let telemetry drive the decision rather than a config edit.
Monitoring your routing
Whatever policy you pick, it is only as good as your visibility into it. Log which model served each request, then watch three signals: per-model error rate, per-model latency, and per-model cost. When a model starts degrading, you want to see it in metrics long before a user reports it.
Choosing a strategy
Start simple and add complexity as you learn your traffic:
- Default to a single well-chosen model per task.
- Add failover behind that default so requests never hard-fail.
- Introduce rule-based routing as distinct task types emerge.
- Only then consider dynamic strategies like least-latency or round-robin, which need telemetry to be trusted.
Most teams can cover the majority of their needs with a default model, failover, and a handful of rules.
Read the docs to see how routing and failover are configured, and browse the model catalog to pick your primaries and backups.