Hardened to carry all your traffic

If you route your traffic through us, our latency and our downtime become yours. This post explains what we built so that trade works in your favor: how we made the gateway fast, what happens when a machine, a provider, or a deploy fails, and how we test the limits. With the code. The API path matters most, so it comes first.

Seven round trips, cut to two

A request through a gateway like ours normally costs seven database round trips before the first token moves: a fail-fast key check, the real authorization, the alias and grant resolution behind your model name, the catalog read behind that, the spend reservation, a route-context write, and the usage settlement. Ours makes two. Almost all of the latency this layer adds is round trips, and we know our count exactly because every request counts its own. Here is how five of the seven disappeared.

1. Caching authorization safely

The first two round trips were the same key lookup, run twice on purpose: a fail-fast check before the gateway decodes your JSON, then the real authorization. Caching authentication is usually a bad idea. We did it anyway, in a narrow way: the second lookup reuses the first for a two second window. The authority cache, from our serving layer:

class AuthorityReuseCache:
    """Short-lived reuse of one key's authenticated authority row.

    Safety: the window only staleness-exposes AUTHORIZATION METADATA.
    Money cannot move on stale authority: gateway_accept_request and
    gateway_start_attempt both re-check revocation/expiry inside
    Postgres at reserve time and fail with 42501."""

    def get(self, key_hash, *, monotonic):
        entry = self._entries.get(key_hash)
        if entry is None:
            return None
        authority, expires = entry
        if monotonic >= expires:      # 2s TTL, then it is gone
            del self._entries[key_hash]
            return None
        return authority

The safety argument is in the docstring. A revoked or expired key can pass the fast check for up to two seconds. It still cannot spend anything, because the reservation re-checks revocation and expiry inside Postgres and fails closed with SQLSTATE 42501. The stale window exposes authorization metadata and nothing else. We cache the cheap thing and re-verify the expensive thing.

2. Caching grants with a written-down bound

The next round trip was the alias and grant join that decides which model your key may reach. It gets the same treatment: GrantResolutionCache, another two second window, on top of an in-memory model catalog. We chose two seconds because it is exactly how long a just-revoked grant could keep serving. The window matches the auth window, so revoking a key or a grant has one worst case, not two.

3. Moving writes off the request path

Two more round trips were writes. Learned route context is display-only metadata that fills two nullable UI columns, so you should not wait on it. A background writer takes it. From the same serving layer:

# Display-only strings must not cost the hot path a round trip: WMO
# calls this between reservation and dispatch, squarely inside the
# pre-first-token window. Enqueue for the background writer; a lost
# write degrades two nullable UI columns, never accounting.
self._ensure_route_context_writer()
with self._route_context_condition:
    self._route_context_pending += 1
self._route_context_queue.put((attempt_id, route_reason, fallback_reason))

Usage settlement moves too: it runs after your response is already streaming. One synchronous write remains, the spend reservation, because it guards real money. On the bring-your-own-key lane there is no money gate to reserve against, so even that write disappears.

4. Keeping connections warm

The last fix was smaller. The edge talks to the worker over HTTP, and httpx closes idle keepalive connections after five seconds by default, so any gap between requests paid a fresh TCP connect. The hop is cluster-local plaintext, so a long keepalive costs nothing. Our client holds connections for a minute:

client = httpx.AsyncClient(
    timeout=httpx.Timeout(_TOTAL_TIMEOUT_SECONDS, connect=_CONNECT_TIMEOUT_SECONDS),
    limits=httpx.Limits(
        max_connections=100, max_keepalive_connections=100, keepalive_expiry=60.0
    ),
)

Together these changes took the worker's own added latency from about sixteen milliseconds to about twelve on a warm machine, cut the p99 under eight-way concurrency by roughly seventy five percent, and more than tripled how many requests a single worker can push.

Those figures describe the Python serving path this design was built in. The hosted data plane has since moved to the open source engine's native Rust implementation, with the Python path kept as a fallback, and the repo now publishes its own measurement: a CI badge showing the p50 of a request through the gateway against a local mock, 20.3 ms as this post is updated. Different measurements answer different questions; the badge is the one you can check without trusting us.

The last hundred milliseconds was geography

After the caching work, the biggest remaining cost was distance. The cluster was in one region and the database in another, so every remaining round trip crossed the country and paid about a hundred milliseconds. No cache helps with that. We moved the compute into the database's region, and the hundred milliseconds became one or two. The move is visible from outside: public time to first byte against the API was 0.6 to 0.8 seconds before the cutover and 0.11 to 0.18 after it.

The move helps more than the API. Member and settings pages make several database reads each, and each read got faster by the same factor. The API is the hot path, but the whole platform sits on the same database.

Built to survive failures

Speed only matters if the gateway stays up. Three things fail in practice: our machines, your providers, and our own deploys. Each has a specific answer.

Machines first. Every service runs multiple replicas behind a load balancer. If the worker that owns your request dies mid stream, a sibling reclaims and settles the work, so a crash does not cause an outage, a half-finished call, or a double charge.

When a provider fails, the request has somewhere else to go. A model routes through a chain that spans more than one cloud: Anthropic direct, Amazon Bedrock, Azure Foundry, OpenRouter, and pooled accounts we hold can all sit in one waterfall. A rate limit or outage on the first route spills to the next, and the same model answers. Failover changes who serves the model, never which model you get.

Deploys are the third failure source, and the most self inflicted one. A new version must pass a readiness check that includes a live database ping before it takes a single request, and services roll one at a time.

Then we tried to break it

We load test until something gives, before real traffic can find the same edge. A load harness in our serving layer fakes the providers entirely: deterministic responses, zero real tokens, zero spend. We throw tens of thousands of requests at the gateway and watch where it bends. Ramped to hundreds of concurrent requests, a single worker does not error or drop. It queues: throughput stays flat while latency climbs, and the queue clears the moment the spike passes. Adding workers brings latency back down. That is the worker's behavior under harness load, and at one point we took our own production gateway down producing it. Every limit we hit that way became one of the guardrails above.

Then real traffic found an edge the harness had not. On August 22 a bulk data import left the database planner with stale statistics. Dashboard reads slowed, requests piled up in the api pod with no bound, and the kernel killed the pod for memory. The public catalog on the website kept refetching from the dead pod, which kept it dead. The failure was queueing without a limit, so the api no longer queues without a limit: past a bound on the requests it is actually working on, 120 by default, it answers 503 immediately, keeps passing health checks, and lets the herd drain. The bound counts in-flight work rather than open connections, so idle keep-alive sockets and long-lived streams do not spend the budget, and the shed path has its own test suite. The harness finds most edges first. This one it missed, and the fix is written down the same way as the ones it found.

Where we are going next

Today the cluster and its primary database live in one region, which survives losing any machine. Next, in order: point in time recovery and a standby database in a second availability zone, then a warm standby cluster in a second region with health checked DNS failover, then failover onto a second cloud, with Azure as that cloud. None of it is live yet. Each item gets added to this post once it runs in production and is measured.

And the strongest guarantee is one a closed platform cannot give you: the gateway engine we host is open source. Audit it, and if your risk model needs it, run the same engine yourself with one command. Provider keys stay at the gateway and never reach the people or agents calling it, whether we host it or you do.