AutoBench Router Benchmark 1 - October 2026
Commercial routers and baselines graded on a frozen answer matrix. Accuracy is the share of questions where the chosen model is statistically as good as the best available answer.
Router BenchmarkQuality versus cost
Dashed lines are Always-Sol, Always-Luna, and Random. Amber points collapsed onto one model.
Accuracy is the share of questions where the chosen model is statistically as good as the best available answer.
Board
| Pass | Vendor | ARI | n | Accuracy | 95% CI | USD / question | Top-model share | Collapsed |
|---|---|---|---|---|---|---|---|---|
| Cheapest-adequate | Baseline | 88.0 | 252 | 100.0% | 100.0–100.0 | $0.0018 | — | |
| Oracle | Baseline | 71.4 | 252 | 100.0% | 100.0–100.0 | $0.0341 | — | |
| Always-Luna | Baseline | 56.1 | 251 | 85.3% | 80.9–89.6 | $0.0043 | — | |
| Always-Sol | Baseline | 48.9 | 248 | 87.5% | 83.5–91.5 | $0.0787 | — | |
| ND k0 | Not Diamond | 48.9 | 248 | 87.5% | 83.5–91.5 | $0.0787 | 100.0% | Yes |
| Azure balanced | Azure Model Router | 47.0 | 179 | 82.7% | 78.8–89.4 | $0.0204 | 48.0% | |
| Azure quality | Azure Model Router | 47.0 | 238 | 85.3% | 80.3–89.1 | $0.0647 | 40.9% | |
| Auto high | OpenRouter | 39.2 | 244 | 79.1% | 73.4–83.6 | $0.0230 | 57.9% | |
| ND k4 | Not Diamond | 39.0 | 249 | 80.3% | 75.5–85.2 | $0.0600 | 77.0% | |
| Auto max | OpenRouter | 37.1 | 240 | 78.3% | 72.9–83.3 | $0.0289 | 78.2% | |
| Auto medium | OpenRouter | 26.7 | 246 | 73.6% | 67.1–78.0 | $0.0131 | 78.2% | |
| ND native | Not Diamond | 21.3 | 250 | 72.0% | 66.0–77.2 | $0.0597 | 63.9% | |
| NadirClaw | NadirClaw | 16.9 | 249 | 70.3% | 64.3–75.9 | $0.0514 | 58.7% | |
| Orca quality | OrcaRouter | 3.5 | 245 | 65.7% | 59.2–71.0 | $0.0005 | 100.0% | Yes |
| Auto low | OpenRouter | 3.5 | 245 | 65.7% | 59.2–71.0 | $0.0006 | 98.4% | Yes |
| Always-cheapest | Baseline | 0.0 | 252 | 44.4% | 38.5–50.4 | $0.0002 | — | |
| Orca balanced | OrcaRouter | 0.0 | 252 | 39.3% | 33.3–45.2 | $0.0002 | 99.6% | Yes |
| Orca cheapest | OrcaRouter | 0.0 | 252 | 39.3% | 33.3–45.2 | $0.0002 | 99.2% | Yes |
| ND k5 | Not Diamond | 0.0 | 251 | 46.6% | 40.2–52.6 | $0.0009 | 82.9% | |
| Brick | Brick | 0.0 | 241 | 63.9% | 57.7–69.7 | $0.0031 | 80.2% | |
| Azure cost | Azure Model Router | 0.0 | 74 | 45.9% | 33.8–56.8 | $0.0091 | 59.1% | |
| Cascade | Cascade | 0.0 | 235 | 60.9% | 53.2–66.0 | $0.0104 | 58.3% | |
| Random | Baseline | 0.0 | 252 | 64.7% | 58.7–70.6 | $0.0115 | 7.1% |
ARI is the AutoBench Router Index at β = 0.2. The board is ordered by it, highest first. It runs from 0 to 100 and is higher when a pass is both accurate and cheap. A pass no better than random scores 0. How ARI is calculated. Accuracy is the share of questions where the chosen model is statistically as good as the best available answer.
Same-pick
Median cross-vendor same-pick is 0 across 59 pairs. The grid is the 12-pass comparison from the report. A cell is the share of questions on which the two passes named the same model.
| Orca cheap | Orca bal | Orca qual | Azure cost | Azure bal | Azure qual | NadirClaw | Brick | Cascade | ND k5 | Auto high | Auto low | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Orca cheap | 1.00 | 0.99 | 0.01 | 0.14 | 0.01 | 0.00 | 0.02 | 0.00 | 0.01 | 0.00 | 0.00 | 0.01 |
| Orca bal | 0.99 | 1.00 | 0.00 | 0.14 | 0.01 | 0.00 | 0.02 | 0.00 | 0.00 | 0.00 | 0.00 | 0.01 |
| Orca qual | 0.01 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.80 | 0.58 | 0.00 | 0.00 | 0.98 |
| Azure cost | 0.14 | 0.14 | 0.00 | 1.00 | 0.19 | 0.03 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| Azure bal | 0.01 | 0.01 | 0.00 | 0.19 | 1.00 | 0.11 | 0.01 | 0.00 | 0.00 | 0.00 | 0.00 | 0.01 |
| Azure qual | 0.00 | 0.00 | 0.00 | 0.03 | 0.11 | 1.00 | 0.12 | 0.00 | 0.00 | 0.00 | 0.01 | 0.00 |
| NadirClaw | 0.02 | 0.02 | 0.00 | 0.00 | 0.01 | 0.12 | 1.00 | 0.00 | 0.00 | 0.34 | 0.00 | 0.00 |
| Brick | 0.00 | 0.00 | 0.80 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.48 | 0.00 | 0.00 | 0.79 |
| Cascade | 0.01 | 0.00 | 0.58 | 0.00 | 0.00 | 0.00 | 0.00 | 0.48 | 1.00 | 0.00 | 0.00 | 0.57 |
| ND k5 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.34 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 |
| Auto high | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.01 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 |
| Auto low | 0.01 | 0.01 | 0.98 | 0.00 | 0.01 | 0.00 | 0.00 | 0.79 | 0.57 | 0.00 | 0.00 | 1.00 |
Scored on the answer matrix published as AutoBench Run 6.