Same model. Real tasks.
Every agent here ran the same model on the same tasks, under each benchmark's own limits. What differs is the agent, and what it pays to get there.
As good as Codex, ahead of Claude Code
On DeepSWE's 113 tasks with DeepSeek V4.1 Flash, Milo and Codex each solved 81 and Claude Code solved 70. Codex and Claude Code ran with their web tools, skills and MCP servers off, so each was down to its own coding tools.
One run moves by about three tasks either way, so Milo and Codex are a tie, and Claude Code's eleven fewer is a real gap.
Up to 50% less. Same work.
An agent re-sends its whole context on every call, so most of what it pays for is cached tokens. Milo routing ranks a model's providers on that real mix and sends calls to the cheapest three, leaving out providers measured to do worse work. AutomationBench, 600 tasks, DeepSeek V4.1 Flash:
| Run | Passed | Cost | Per task |
|---|---|---|---|
| Milo routing | 342 / 600 (57.0%) | $24.17 | $0.040 |
| OpenRouter's default routing | 348 / 600 (58.0%) | $47.25 | $0.079 |
| Public score for the model | 54.8% | — | — |
Both Milo runs used the same model and limits; only the routing changed. Cost is what OpenRouter billed. The public score is the model in AutomationBench's own harness.