Engineering
How the gateway is built: the 1 ms hop, provider failover, and the routing policy under load.
How we make the API fast
An inference gateway is one hop in front of the model, and a naive one charges tens of milliseconds for it. Ours charges about one. Sitting next to Postgres, one database call before dispatch, a Rust data plane, an async ledger that took a worker from tens to hundreds of requests per second, and the $0 load rig that keeps the numbers honest.
10 minFailing over LLM providers without changing the model
How another authorized provider can serve the same model when a route fails. Which failures allow fallback, how streaming commits a route, how observed and estimated usage affect accounting, and why prices, capacity, and deadlines still matter. From the open-source engine.
12 minHardened to carry all your traffic
How we decide when to retry a provider, switch routes, keep useful cache, or stop. Explore the routing policy, see how token counts and nested fallbacks work, and read the production findings and test results.
55 min