Do LLM Routers Pay for Themselves?
A pre-registered evaluation of 17 router configurations on long, open-ended questions at billed prices (White Paper)
Router Benchmark
We graded 17 router configurations on 252 long, open-ended questions at billed prices. None beat the best low-cost fixed model on both quality and cost: always using GPT-5.6 Luna scores 85.3% at $0.0043 per question, and no configuration was both better and cheaper across 10,000 resamples. Five configurations send 90% or more of questions to one model. A much larger saving was available and was not realised. Early AutoBench Beta Router results sit above the best fixed model at the same cost; independent validation is still in progress.
Autobench.org team•23 October 2026
Download the file↓