Research — Experiential Labs

June 2026 · ICML Spotlight

Size Doesn't Matter: Cosine-Scored Sparse Autoencoders

Silen Naihin, Lev Stambler

We identify a limitation in how sparse autoencoders detect features through inner product scoring — the approach conflates directional alignment with input magnitude. Since sublayer normalization already removes magnitude information that models don't utilize, we propose replacing inner product with a learned combination of cosine similarity and input magnitude. Our method allows each feature to independently determine how much norm-dependence to employ. The cosine-scored approach never recovers full inner product scoring and identifies features aligned with human-recognizable concepts more frequently, despite comparable reconstruction. Cosine scoring should be the default for dictionary learning on normalized representations.

arXiv

June 2026

CLaaS: Continual Learning as a Service for Sample Efficient Online Learning

Kion Fallah, Silen Naihin, Barak Widawsky, Qingqing Mao

We investigate online continual learning for deployed LLM agents that must adapt to distribution shift. We propose CLaaS, a system which enables agents to improve during deployment, abstracted behind a chat API. To boost sample efficiency, CLaaS stores rollouts in an experience replay buffer for gradient reuse during asynchronous training. We evaluate on an adversarial task, finding that parametric updates lead to superior forward transfer and less forgetting than in-context learning.

arXiv