For the full data, visit autobench.org or our Hugging Face Leaderboard.
Figure 1. Best Answer rate (Delta-Hit) against billed answer cost. The prize is the region above the random quality line and left of the random cost line. Always-Luna already sits there as a static policy. No commercial router does. Points on the random line at a very low price are collapsed cheap models, not a capture of that prize.
Is your router sending each prompt to the right model, or is it a knob in front of a constant? At AutoBench we are very interested in routing, so we built the Autobench Router Benchmark (ARB) to find out. Using the Autobench LLM evaluation methodology, we built a full set of prompts and scored answers across 20 LLM models. Then we tested each router’s model choice against the scores. Seven of the most common commercial and open-source routers were analyzed. None of them beats always calling one mid-priced model on quality and cost together.
Key findings
- The prize is real. A policy that knows, for each prompt, the cheapest model still inside the quality bar scores 100% Delta-Hit at $0.0018 per question. Always calling the top model (GPT-5.6 Sol in our pool) scores only 87.5% at $0.079.
- Capture is zero. Seventeen headline settings from seven systems leave the region above random quality and cheaper than a random pick empty of commercial routers. A mid-tier model like Gpt 5.6 Luna already sits there as a static policy. In 10,000 bootstrap resamples, no setting ever beats it on both quality and cost.
- Five of the seventeen settings are constants. One model takes at least 90% of their decisions. A quality knob often just changes which constant you get.
- Independent routers do not agree. Across 59 cross-vendor pairs, the median share of prompts where two routers named the same model is 0.
Here is who falls short and how, and how the benchmark works.
The promise of routing: sending each prompt to the best LLM
A router promises two things at once: quality close to or better than the best model, and a bill close to the cheapest model that could have answered. You can already get each half without a router. We score quality with Best Answer Rate, also defined as Delta-Hit, the share of questions where the model a router picked was statistically as good as the top models on that question.
The prize
The prize is real. A per-prompt oracle that always names the best model in the pool is perfect Delta-Hit (100%) at $0.034 per question. A policy that picks, for each prompt, the cheapest model still inside the quality bar is also perfect, at $0.0018. Always calling GPT-5.6 Sol costs $0.079 and scores only 87.5%. In other words, the perfect router offers the potential to gain about 12 points in quality against the best static model, while costing about 98% less. That gap is what routing is supposed to collect.
The bar to beat
The board marks the prize as the region above random quality and cheaper than a random pick (64.7% at $0.0115). Always calling GPT-5.6 Luna already sits there: 85.3% at $0.0043, most of Sol’s quality at about 5% of the bill, and no router required. A commercial router earns its keep only if it enters that same region — and, to be worth paying for, if it also beats Luna on both axes. Until this benchmark, we had no independent, cost-aware way to test both promises on long, open-ended prompts that cost what an enterprise prompt costs.
What we found
We called each router between 30 August and 1 September 2026, as shipped or as documented. Across 17 settings the picture is the same.
The prize quadrant is empty
None of the routers occupy the prize region in Figure 1. None beat the quality of the top model (GPT-5.6 Sol) in the pool, let alone at a lower price. None beat Always-Luna on quality and cost together. We resampled the data 10,000 times. No setting ever did. The cheap settings that look as if they enter the region sit on the random line, because they have collapsed onto one flash model. The strong settings sit above random because they buy a strong model, and they cost more than random does. No pass is both.
Two settings reach the neighbourhood of Always-Sol. Not Diamond at knob 0 scores 87.5% at $0.0787 — by naming Sol on every prompt. Azure quality scores 85.3% at $0.065, inside a 3-of-20 catalog. That is Luna’s quality at fifteen times the price.
The money split says the same thing. Every named answer is a hit or a miss, and cheap or costly. Cheap routers land a large share of prompts in “cheap miss.” Strong routers land in “costly hit.” The cell that would justify a router — a cheap hit — stays a minority.
Figure 2. Where each scored prompt landed. Azure quality and Not Diamond at knob 0 are almost all costly hits. Orca’s cheap settings and Auto low buy cheap misses. Nobody fills the cheap-hit cell.
Most routers are constants with a knob
Five of the 17 settings send at least 90% of prompts to one model, so their quality score is just that model’s score. On Orca and on OpenRouter Auto, the knob changes which constant that is.
Not Diamond’s cost–quality knob is a cliff, not a dial. Quality holds through knob 4 (80.3% at $0.060). At knob 5 the cost falls about 64-fold, and so does quality: 46.6%, below random. There is no useful middle.
Figure 3. Not Diamond’s knob is a cliff between knobs 4 and 5. The left side matches a strong static policy. The right side is worse than a random in-pool draw.
Independent routers do not agree
If there were a discoverable right model per question, independent routers would converge on it. They do not. Across 59 pairs whose two members come from different vendors, the median same-pick rate is 0, and 57.6% of the pairs are exactly 0.
Where the rate is high, a shared collapse explains it. Orca quality and OpenRouter Auto at the low tier agree on 248 of 252 prompts (rate 0.98) because both default to the same cheap flash model. The one agreement that is not a shared collapse is NadirClaw against Not Diamond at knob 5 (rate 0.34). Both are classifiers doing per-prompt work. Neither beats random on the outcome.
Figure 4. Agreement among twelve passes. The diagonal is 1. Bright off-diagonal cells are shared collapses. The dark field is the usual case: two routers do not name the same model.
One number: the Autobench Router Index
Delta-Hit and dollars sit on different axes. The Autobench Router Index (ARI) answers the joint question at a locked cost weight β = 0.2. Quality is how far signal Delta-Hit sits above the frozen random policy. Cost is a log score between two fixed dollar anchors. A pass that does not beat random on quality scores ARI 0, no matter how cheap it is.
At this weight, Always-Luna is first. No commercial pass has positive lift against it. Not Diamond at knob 0 matches Always-Sol. Every pass that sits on or below the random quality floor scores 0.
Figure 5. Official Autobench Router Index at β = 0.2. Black bars are Always-Luna and Always-Sol. Blue bars are commercial passes above zero. Always-Luna leads; no commercial pass has positive lift.
Router by router
Several systems could only reach a slice of our 20 models (Not Diamond 7, NadirClaw 4, Azure 3, Brick 3), so read the notes below as how they behave inside that slice, not as a ranking.
- Not Diamond (hosted, cost–quality knob 0 to 10) is a cliff, not a dial. At knob 0 it sends every prompt to Sol, so you pay Sol’s price. Quality holds to knob 4 (80.3% at $0.060). At knob 5 it falls to 46.6%, below random.
- OpenRouter Auto (hosted, four cost tiers) is a spend tier, not a route. The low tier sends 98% of prompts to one cheap model and lands on the random line (65.7%). The high and max tiers (79.1% at $0.023 and 78.3% at $0.029) score alike, and max costs more.
- Orca (hosted, named strategies over a model list you supply) lets the knob pick the constant. Its cheapest and balanced strategies send 99 to 100% of prompts to one open model and score 39.3%, worse than random. Its quality strategy sends nearly everything to a cheap model instead (65.7%).
- Azure Model Router (Microsoft’s router, inside Azure’s own deployments) gives strong quality at a strong bill. Its quality option scores 85.3% at $0.065: Always-Luna’s quality at fifteen times the price.
- NadirClaw (a small local classifier that sorts each prompt into a tier of models) sent most prompts to Sol and scored 70.3% at $0.051, close to random.
- Brick (matches each prompt against a map of model capabilities) scores 63.9% at $0.0031, on the random line. We ran it as shipped, on models it was not tuned for, so this shows how it copes with a new pool, not its best case.
- A FrugalGPT-style cascade (tries a cheap model first and escalates when a check fails) scores 60.9%, below random.
How the AutoBench Router Benchmark works
ARB extends the AutoBench method: fix the test, then grade each router by the model it picks.
- A fixed test of open-ended questions. 252 prompts across ten domains and five difficulty levels — long, open-ended items, not short multiple-choice. Twenty models answered each one once, and a committee of ten LLM judges scored every answer — the Collective-LLM-as-a-Judge approach behind AutoBench. The real billed cost is stored next to each score. Those answers cost on the order of 10–100× more than a boxed-letter item, which is the regime a router is sold into.
- Grading by lookup. A router only names a model. We read that model’s stored score and cost, so a router cannot improve its result by writing a better answer.
- Rules locked first. We wrote down the scoring rules on 29 August 2026, before any router was called.
- Behaviour, not just quality. Next to Delta-Hit and cost we report collapse share (does one model take ≥ 90% of decisions?) and same-pick rate (do two routers name the same model?). A constant cannot hide behind a single score.
- Cheap to extend. The answers and judgments are paid for once, so testing a new router is a lookup.
Delta-Hit is not exact-match accuracy. A pick is a hit when its stored score lands within a locked window of the top three in-pool models on that question. Read it against the random line (64.7%), not against 100%. The board is packed near the top of the scale, so a coin-flip already looks decent.
What came before
RouterBench (Hu et al., 2024) released more than 405,000 model outcomes so that routing algorithms can be compared offline on a fixed pool — pay for the answers once, then score any policy by lookup. RouterArena (Lu et al., 2025) runs a live public leaderboard of academic and commercial routers on short multiple-choice questions, ranked by a score that combines accuracy and cost.
ARB keeps the lookup design and changes the question. AutoBench items are long and open-ended. They look like the prompts a buyer actually sends, and they cost one or two orders of magnitude more to answer than a boxed letter. On short multiple-choice boards a cheap model is often already the best choice, so a router that always names it can look strong. On this board the same products tell a quieter story: a quality knob often collapses onto one frontier model, a cheap tier sits on the random line, and two independent vendors almost never name the same engine for the same prompt. Collapse share and same-pick rate sit next to quality so that pattern cannot hide inside a single score.
Before you buy a router
Five questions to ask before you pay for one.
- What is the static policy? If the router cannot beat always calling the best mid-priced model in the pool, on your prompts, it is a knob in front of a constant.
- What is the top-model share? At 90% or above, the quality number is that model’s number. Ask for the share at every knob, not only the default.
- Does the knob have a middle? A cliff from “always strong” to “worse than random” is not a cost–quality curve.
- Do two independent routers agree on your prompts? If they agree only because both default to the same cheap model, neither has found a right answer.
- Which board was the claim measured on? Exact-match accuracy on short items is not Delta-Hit on open-ended prompts that cost one or two orders of magnitude more to answer.
What's next
Scoring another router on this board is a lookup. Adding a question is the expensive step, because every new question has to be answered by the pool and judged. The next increment of scientific value is therefore a larger corpus — with paraphrase sets built in, and with room for per-domain and agentic slices — not a longer roster of vendors on the same 252 questions. The roster can grow once the corpus exists.
The prize is real, and we think it can be collected. The Beta Router is live: Try an LLM Router That Actually Routes.
Join the Community
The white paper is Do LLM Routers Pay for Themselves?, by Autobench.
AutoBench is a step towards more robust, cost-aware, and future-proof LLM evaluation. We invite you to explore the data, read the paper, and join the discussion:
- Read the paper: Do LLM Routers Pay for Themselves?
- Use the data: the questions, the score-and-cost matrix, and the decision logs are released as a Hugging Face dataset. You can recompute every table without calling a router again. Router Benchmark Dataset
- Explore the leaderboard: AutoBench-Leaderboard on Hugging Face, and the live board on autobench.org
- Request a router benchmark on your own traffic: use the contact form on autobench.org with the routers and the domain you want tested
- Tell us what to add: the larger board needs a deliberate pool decision and paraphrase sets built in. Send us the routers and workloads you want on it
- Follow along: autobench.org · X @AutobenchAI · LinkedIn
We strongly encourage the AI community to engage with the interactive leaderboard, explore the released data, and share feedback.