97% cheaper than Opus 4.8, and 50% better than baseline.
We trained a macOS computer use agent to help a customer get off a frontier API and onto a 9B model. When evaluated in closed-loop simulation on tasks resembling production, we measured a +50% gain from training.
- Speed
- 7× faster than Opus 4.8
- Cost
- −97% vs Opus 4.8
- Quality
- +50% over baseline
The task, and what it costs
A computer use agent operates a real desktop. It reads an accessibility tree, the structured hierarchy of windows, buttons, fields and text that macOS already exposes to screen readers, then decides what to do next: click a control, type into a field, open a URL, quit an app.
A single window's tree runs to hundreds of nodes and thousands of tokens, and the agent has to re-read it after every action. Token spend therefore scales with how complicated the interface is and with how many steps the task takes. Running with Opus 4.8 at high volume becomes infeasible due to the token costs.
Distillation for efficient training
Training a small model for this needs examples of work being done well. We generated a list of 1,000 tasks using apps like Safari, Notes, Calculator, Activity Monitor, and others. We used Kimi K2.6 as a strong teacher to collect high quality data, driving fleets of Mac VMs. Every step was captured as a training example: the full accessibility tree the teacher saw, reasoning in the model's chain-of-thought, the action it chose, and what happened next.
Not everything the teacher produced was worth learning from. Trajectories were evaluated using an LLM judge and filtered down to the ones that actually completed their task. The remaining tasks were then distilled into Qwen 3.5 9B.
Closed-loop evaluation in simulation
Understanding how model training transfers to production requires running tests with the same tools and accessibility tree that it would see in production. Since agents are non-deterministic, we are nearly guaranteed that the model's actions will deviate from the trajectories collected on the Mac VMs. We were able to bypass this by building simulations within our open source tool, using a RAG database over our collected Kimi K2.6 traces. When running tests, an LLM acts as a text world model, predicting how the accessibility tree will evolve for each model action.
When measured against these simulations, on the same tasks and the same rubrics, the trained 9B model completes 50% more of them than the untrained Qwen 3.5 9B it started from, while costing 97% less to run than Opus 4.8. Because the simulation is rebuilt from the real environment rather than hand-written, retraining and rescoring is cheap enough to do repeatedly instead of once.