Writing
Notes from Experiential Labs on the gateway, reliability, and the intelligence layer.
How we make the API fast
An inference gateway is one hop in front of the model, and a naive one charges tens of milliseconds for it. Ours charges about one. Sitting next to Postgres, one database call before dispatch, a Rust data plane, an async ledger that took a worker from tens to hundreds of requests per second, and the $0 load rig that keeps the numbers honest.
10 minReliability · August 2026Failing over LLM providers without changing the model
The same model, sold by several providers, should survive any one of them failing. Which failures are allowed to spill, how the waterfall keeps model identity fixed by construction, what streaming and billing do under failover, the gaps that remain, and how we hardened the rungs once failover shipped. From the open-source engine.
11 minReliability · August 2026Hardened to carry all your traffic
A request through a gateway like ours normally costs seven database round trips. Ours makes two. The tricks that got it there, the region move that fixed the rest, failover that spans clouds, and the time we took our own gateway down on purpose. With the code.
11 minPaper · June 2026Size Doesn't Matter: Cosine-Scored Sparse Autoencoders
Inner product scoring conflates directional alignment with input magnitude. We propose cosine scoring for dictionary learning on normalized representations. ICML Spotlight.
2 minPaper · June 2026CLaaS: Continual Learning as a Service for Sample Efficient Online Learning
Online continual learning for deployed LLM agents behind a chat API: experience replay for gradient reuse, superior forward transfer, and less forgetting than in-context learning.
2 min