The same model is often sold by several providers. fable-5 is available from Anthropic directly, on Amazon Bedrock, and through OpenRouter. Each of those routes has its own rate limits, its own credits, its own availability, its own latency, and its own error formats. So a provider outage should slow you down at worst. It should never make the model unavailable. This post describes how our gateway does that, with the code.
Retries are not enough
When a provider call fails, there are three possible responses. You can retry the same provider, which works for a blip and does nothing for an outage: your retry dies with the provider. You can fail over to the same model on a different provider, which is what this post is about. Or you can silently switch to a different model, which is the dangerous one: the caller asked for a specific model, tested against it, and tuned prompts for it. Swapping it under them changes behavior without telling anyone. The gateway does the second and is built so the third cannot happen by accident.
Which failures are allowed to fail over
The rule the code enforces: a request spills to the next provider only when the failure belongs to the provider binding, never when it belongs to the caller's request. Provider-binding failures are 429s, 5xx, timeouts, connection failures, and streams that end without a terminal event. They also include provider-credential 401s and 403s and provider-side 404s. That is broader than capacity failures on purpose: a misconfigured credential or a renamed model ID on one provider says nothing about your request, and another rung can serve it. Caller failures, meaning every other 4xx, return a corrective 400 and are never counted against the provider's health. From exp/runtime/models/providers/errors.py:
def _transport_failure(status_code: int | None) -> GatewayFailure:
"""Classify one sanitized HTTP or connection failure by status only.
Remaining 4xx statuses are the caller's request being rejected, so
they map to INVALID_REQUEST: the caller gets a corrective 400 and the
deployment health circuit never counts the attempt against the
deployment."""
if status_code in {401, 403}:
return GatewayFailure(
failure_class=GatewayFailureClass.PROVIDER_AUTHENTICATION,
safe_message=(
"provider authentication failed; ask the gateway operator "
"to verify the provider connection credential"
),
failover_eligible=True,
)
if status_code == 429:
return GatewayFailure(
failure_class=GatewayFailureClass.THROTTLED,
safe_message=(
"provider throttled the request; retry after the delay "
"in the Retry-After header"
),
failover_eligible=True,
)Refusals do not fail over by default. A model declining a request is model behavior rather than a provider fault, and the next rung serves the same model. There is an opt-in flag for teams that want refusal failover anyway; it is covered below.
The waterfall
Each rung of a model's chain is a deployment: one way to reach one exact model. For fable-5 the chain is Anthropic on your key, then Bedrock on pooled accounts, then OpenRouter on your credits. An ordered set of deployments is an ExactModelPool, and the pool is where model identity is enforced. Every deployment in a pool carries the same exact_model_id, and the schema refuses to build a multi-deployment pool unless an operator has certified the deployments as equivalent. From exp/common/models/gateway_catalog.py:
class ExactModelPool(ContractModel):
"""An ordered set of deployments certified as one exact logical model."""
pool_id: ExactModelPoolId
exact_model_id: ExactModelId
deployment_ids: tuple[DeploymentId, ...] = Field(min_length=1)
equivalence: GatewayEquivalenceCertification | None = None
@model_validator(mode="after")
def _require_unique_deployments(self) -> ExactModelPool:
if len(set(self.deployment_ids)) != len(self.deployment_ids):
raise ValueError("exact-model pool deployments must not repeat")
if len(self.deployment_ids) > 1 and self.equivalence is None:
raise ValueError(
"multi-deployment pools require operator equivalence certification"
)Failover picks the next deployment from the same pool, so a rung cannot point at a different model. The routing code states the contract in its own words: choose a safe retry or later exact deployment without changing logical model.
Failover during streaming
Streaming constrains failover, and the boundary is the first semantic event. Until a text delta, a tool call, or a visible refusal has reached the client, the gateway can retry the same rung or advance to the next one freely. Sequence numbers are rewritten into one public sequence, so a provider swap before that point is invisible on the wire. The first semantic event commits the route. After that, failover is off: if the provider dies mid stream, the request ends as a truncated stream. There is no resume and no silent restart, because a restart could replay or reorder tokens the client already has.
The opt-in refusal flag works inside the same boundary. With it set, refusal deltas are buffered rather than sent, bounded at 64KB or 256 events, and the route advances to the next rung without the refused output ever reaching the client.
Billing through failures
Every physical dispatch is reserved in a durable ledger before the provider is called, at a conservative worst-case cost, and settled exactly once afterward. The settle-once path is tested end to end against the failure cases that produce double charges elsewhere: an empty completion settles at zero, a truncated completion settles at exactly the delivered tokens, a provider error after retries settles at zero with the reservation released, and a post-commit crash with no usage received settles at zero.
Caller retries have their own contract. /v1/chat/completions and /v1/responses accept an Idempotency-Key header. The same key with different content is a typed conflict. The same key with the same body replays the completed response exactly, from a bounded in-process replay store; concurrent callers wait on the owner and get the same result, and a failed owner is abandoned fail-closed rather than retried under the caller's key. The durable ledger itself stores a hash of the request and no content. Replays never touch the ledger or the budget, so a replay cannot double a charge. Each rung also derives a stable idempotency key for its upstream call, reused on same-rung retries and fresh for each new rung; whether the provider honors it is up to the provider.
Tool calls across providers
Failover assumes the next provider produces the same wire behavior, and by default it does not: tool calls and structured outputs differ across the OpenAI, Anthropic, Bedrock, and Gemini wire formats. The engine normalizes every provider into one internal event stream, with per-provider mappers on the way in, and re-encodes that stream into the caller's dialect on the way out, whether the caller speaks OpenAI Chat Completions, OpenAI Responses, or Anthropic Messages. A dated certification matrix records which capabilities are proven per provider, and its cells admit unproven states, including untested for lack of credentials.
Knowing a provider is down
Health tracking runs in process with no I/O, so consulting it costs nothing on the request path. Two consecutive operational failures open a deployment's circuit for thirty seconds, followed by one half-open probe. Throttling gets its own separate thirty second window, so a 429 storm does not open the failure circuit. Provider-credential failures and 404s suppress a deployment immediately. And a fully suppressed route still admits bounded last-resort probes, so a model never fails purely on circuit state while a provider has already recovered.
When credits run out
Running out of credits looks like a failure but routes differently, and the scope decides. Exhaustion scoped to one deployment advances the waterfall to the next rung. Exhaustion scoped to the organization, a key, or a model stops routing and returns 429 insufficient_quota: the gateway never silently moves your traffic onto a paid lane you did not choose. Bring-your-own-key traffic never reaches the credit gate at all.
What the ledger keeps
All of the above is observable without retaining content. Both ledgers are content-free by contract: per-attempt rows carry the provider, the exact model, the route depth, the failure class, token counts, and cost, and there is no prompt or response column anywhere in the schema. The idempotency store keeps only a hash of the request. A test plants secret canaries in requests and asserts they persist nowhere.
The gaps
Five things are weaker than the design above suggests. Equivalence across providers is operator-attested configuration, dated and carrying an evidence hash; nothing samples two providers and compares outputs. Customers see the final provider and an attempt count, while the full ordered attempt trail is recorded internally but not yet exposed. Quota rejections lose their specific reason at the HTTP boundary, so the response carries a generic quota message even though the ledger knows which limit fired. Non-streaming requests currently ride the streaming path, which re-pays the provider's inter-token pacing; that is a known performance cost. And the chaos suite faults one deployment at a time; multi-provider failover under load has functional coverage and no load drill.
The failover engine described here, the classification, the pools, the streaming boundary, the ledger, the health circuits, is open source in experientiallabs/experiential, first shipped in release 0.5.0 with 0.5.2 current, and it is the same engine we host: since 0.5.1 the hosted data plane runs the repo's native Rust implementation, with the Python engine as its fallback.