Model providers are infrastructure, and infrastructure fails. Single-provider outages are common enough that any production app should treat "the API is down" as an expected event, not an emergency. The tool for that is failover.
The failure modes you should plan for
Not every failure is a full outage. The realistic list includes:
- A hard provider outage (5xx errors, total unavailability).
- Rate limiting or quota exhaustion on your account.
- Slow or degraded responses that time out.
- A model being deprecated or silently retired.
Each of these can take a user-facing feature down if you have no fallback path.
How automatic failover works
Instead of your app catching an error and giving up, a gateway with failover does the retrying for you:
- Your request arrives at the gateway.
- The gateway routes it to the primary provider.
- On failure or timeout, it retries on the next available provider or model.
- You receive a normal response; the failure never reaches your app.
The key property is that your app does not need to know which provider served the request — it just gets an answer.
What makes failover actually good
A naive retry is not enough. Reliable failover needs a few specific behaviors:
- Retry budgets — cap the number of attempts so a total outage fails fast instead of hanging.
- Different-provider fallback — retrying the same provider three times does not help if the provider is down.
- Timeout handling — fail over on slow responses, not only on hard errors.
- Idempotency awareness — avoid double-charging or double-executing side effects on retried requests.
A well-behaved gateway exposes these knobs so you can tune retries per request.
Building a failover ladder
The most resilient setup is a ladder, not a single backup. Rank your candidate models from preferred to last resort, and let the router step down one rung at a time. A common ladder: your best-quality model first, a cheaper equivalent second, and a fast fallback third — so you keep serving requests even when the top two providers are both struggling. The fallback rungs should already be configured and tested, because the moment a provider goes down is a bad time to start wiring a backup.
Health checks and circuit breakers
Failover is faster when the gateway already knows a provider is sick. Health checks poll providers for errors and latency, and a circuit breaker takes a failing provider out of rotation for a cool-down period instead of letting every request wait on it and time out. The combination turns a provider outage from a series of slow failures into a fast, clean switch.
Design so failure is invisible
Even with failover, your own code should be defensive:
- Treat any request as potentially retried, and keep retry logic idempotent.
- Set client timeouts so your app does not wait forever on a hung upstream.
- Log the model and provider that actually served each request for debugging.
With those habits, a provider outage becomes a latency blip in your metrics rather than a support ticket.
What high availability does not solve
Failover keeps requests flowing, but it does not make a degraded provider fast or a wrong answer right. If a fallback model is lower quality on your task, the answer you get during an outage may be worse — just not absent. Budget for that trade-off when you choose your ladder.
Mappace fronts every provider behind one endpoint, so failover is a routing concern rather than an integration effort. Read the docs to see request-level routing options, and check the model catalog to pick the fallback models you want on deck.