A gateway is a hop in front of the model. You call us, we authenticate you, pick a route, and call the provider on your behalf. Every millisecond we spend doing that is a millisecond you pay before the model produces anything, on every single request. Naive gateways add tens of milliseconds of their own. We think the only honest target is the noise floor: the gateway's overhead should be small enough that what you pay for is the model, and effectively nothing else.
On our fastest lane we are at about one millisecond. This post is the story of getting there: where the time actually goes, and the specific changes, a region move, a series of round-trip cuts, a Rust data plane, an async ledger, and a rebuilt edge, that each removed their share.
Where the time goes
Start by writing down everything that must happen to one request before the first token can move. We authenticate your API key. We resolve the model name you sent into a concrete route, which means alias and grant resolution against the catalog. If the request is platform-funded, meaning it spends our money with the provider rather than yours, we reserve budget for it before we dispatch. Then we dispatch to the provider and stream the response back. After your stream ends we settle: count the tokens, price them, record usage.
The first four steps are the hot path. Settlement is not, or at least it does not have to be, and a lot of this post is about making sure work that can happen after your tokens does happen after your tokens. The other thing to notice is which steps naturally live in the database. Authentication, resolution, reservation, and settlement all do, and each database round trip has a fixed physical cost. Round trips, not CPU, are the budget.
Physics first: sit next to your database
Before any clever engineering, geography. The gateway used to run in a different AWS region than its Postgres database, and a cross-region round trip costs about 100 ms. It does not matter how few queries you make when each one pays that. We moved the gateway to sit next to Postgres in us-west-2, and a typical query went from about 100 ms to about 1 to 3 ms, roughly 13x. End-to-end time to first token through the gateway went from about 0.75 seconds to about 0.15. This is live in production today.
It is a little humbling that the largest single win in this story involved no code. If your servers and your database are in different regions, nothing else on this page matters yet. Fix that first.
Then make fewer round trips
Once each round trip costs single-digit milliseconds, the count of round trips becomes the whole game. Done naively, the request anatomy above costs seven database calls on the hot path. Ours makes two, and after the last change below, the measured average on the platform-funded lane is just above one. Four changes got it there.
First, caching the reads. The authority behind your key (is it valid, what may it do) and the alias and grant resolution behind your model name are reads that almost never change between two requests arriving milliseconds apart. Both are cached in process with short TTLs. The reservation that actually moves money is never served from a cache; a stale entry can tell us who you probably are, it cannot spend anything.
Second, moving writes off the request path. Usage settlement runs after your response is already streaming, and learned route context, which fills display-only fields, goes through a background writer. Between the caching and the deferral, seven hot-path database calls became two.
Third, one round trip per call. A single-statement write wrapped in an explicit transaction costs three round trips: BEGIN, the statement, COMMIT. Postgres commits a lone statement atomically on its own, so each single-statement money call now commits in the same round trip that runs it. Same guarantees, a third of the trips.
Fourth, the fold. The two calls left before dispatch were recording the accepted request and reserving budget for it, run back to back against the same database. They are one call now. Measured across the load rig, the platform-funded lane went from 2.0 database calls per request to 1.05, and the granted bring-your-own-key lane from 3.0 to 2.02.
One engine, two lanes
The gateway wraps our inference engine, experiential, in process; the platform does not reimplement serving logic on top of it. The engine ships a compiled Rust gateway server that owns the public port. Whatever the Rust server does not natively serve, it forwards to the Python control plane behind it, so one port presents one API regardless of which lane answers.
Per request, the lane is chosen deterministically. Unkeyed bring-your-own-key traffic is served by the Rust native lane. Platform-funded traffic, and keyed or replay traffic, runs on the Python execution path, which carries the budget accounting and cross-worker replay.
The split is not really a Rust-versus-Python performance argument. The guarantees that make the platform lane trustworthy, fail-closed budget reservation and replayable state, live in Postgres and in the Python code that talks to it. The Rust lane exists for the traffic that needs none of that: stateless requests that want authentication, routing, dispatch, and nothing else. So the stateless path gets a compiled data plane, and the stateful path stays where the state is.
On the same box, against a mocked provider, the native bring-your-own-key lane measures about 3.36 ms to first token, and the Python lane about 3.6 to 4.1 ms. The gap is real but small, which we read as the Python path spending its extra time on honest work rather than overhead for its own sake.
The throughput unlock: an async ledger
Latency is what one request feels; throughput is what the fleet costs. The single biggest capacity win in the gateway's history was not a cache or a rewrite. It was making the ledger asynchronous.
The engine's ledger calls, the database writes that account for each request, used to be synchronous calls running on the worker's single async event loop. A synchronous call on an event loop does not just slow its own request down. It stops the loop, so every in-flight request waits behind it. The symptom set is distinctive once you have seen it: under load, one CPU core pegged around 91 percent while Postgres sat near 3, throughput flat no matter how much concurrency we offered, and time to first token growing linearly with load. The worker had become a single-file queue in front of an idle database.
Moving the ledger to an async path lets the worker and Postgres work in parallel. On the mocked load test, a single worker went from about 25 to 35 requests per second to about 740 to 805 at 64 to 128 concurrent, with p99 time to first token between 135 and 154 ms, zero errors, and no 5xx cliff anywhere up to 256-way concurrency. Integrity held too: across the tens of thousands of requests in those runs, none was left unsettled in the ledger. And because this is a per-worker ceiling, replicas multiply it.
The edge stops being the bottleneck
In front of the worker sits the public /v1 edge, a thin proxy that terminates the outside world and forwards to the worker. Thin is not the same as free, and under load the edge was the collapse point. Four changes fixed it. The model list is cached at the edge, so listing models skips the worker hop and the re-serialization entirely. The edge-to-worker HTTP client moved from httpx to aiohttp. The event loop and parser moved to uvloop and httptools. And the auth middleware was rewritten as pure ASGI, which removes the per-request object churn a framework middleware stack pays.
At 32 concurrent, the edge went from 235 to 736 requests per second, about 3.1x, and median latency from 112 ms to 41 ms. The comparison had a control built in: the one lane these changes did not touch, the Rust-served path, stayed flat across the A/B. That is how we know the rig measured code and not noise.
Two cost lanes, one honest difference
Everything above applies to both ways of running through us, but the two lanes are not identical, and we would rather explain the difference than blur it. A bring-your-own-key request uses your provider key. We never spend our own money on it, so there is no platform budget accounting anywhere on its hot path, and it rides the fastest lane at about 1 ms of gateway overhead.
A platform-funded request spends our money with the provider on your behalf, and before we do that we make a synchronous, fail-closed budget reservation. If the budget is not there, the request fails before a token is bought, not after. That reservation is the one structural step this lane has and the other does not, about 1 to 3 ms colocated, and the fold above cut it to a single pre-dispatch database call. We consider the trade correct: the alternative to reserving before you spend is discovering the problem after you spent.
How we measure without spending
Every load number in this post comes from a rig designed to cost zero dollars. The provider is mocked by a loopback responder: no real model is called and nothing is spent, which is exactly what you want when the question is how much overhead the gateway itself adds. Provider time is real and separate; these numbers deliberately exclude it.
The rig runs a concurrency ladder from 8 to 256. One detail matters more than it looks: the load is sharded across multiple client processes. A single async client process saturates itself somewhere past 16 concurrent streams, and from that point on it measures its own ceiling, not the server's. Run one client and you understate the gateway by several times while the numbers look perfectly plausible. Shard the clients and the server becomes the thing being measured.
The floors
Stated plainly. A warm request that does zero database work spends about 0.9 ms in our code. A bring-your-own-key request is about 1 ms. A platform-funded request is about 2 to 4 ms on a local dockerized box, and lower colocated in production, where the database is 1 to 3 ms away. All of these are gateway-overhead numbers against a mocked provider; the model's own time sits on top, unchanged by us.
So the name of this page is a real number with a real scope. One millisecond is the fast lane and the application floor, and the platform lane sits just above it, paying only for the guarantee it gives you. That is the goal in one sentence: the model's time should be what you pay for, and the gateway should be the part of your stack you never have to think about.