Back to Archive

AutoBench Router Benchmark 2 - November 2026

November re-audit on the same frozen answer matrix. Adds OpenRouter's rebuilt Auto Router, Conifer, and dari (select-only). Accuracy is the share of questions where the chosen model is statistically as good as the best available answer.

Router Benchmark
Date
November 17, 2026
Version
2026-11-17
Routers graded
26
Baselines
6

Quality versus cost

Dashed lines are Always-Sol, Always-Luna, and Random. Green points are AutoBench. Violet is OpenRouter's November Auto Router. Blue is Conifer. Orange is dari. Grey points are October when that board includes them. Amber points collapsed onto one model. Whiskers on November points are 95% bootstrap intervals.

Accuracy is the share of questions where the chosen model is statistically as good as the best available answer.

Board

PassVendorARInAccuracy95% CIUSD / questionTop-model shareCollapsed
Cheapest-adequateBaseline88.0252100.0%100.0–100.0$0.0018—
OracleBaseline71.4252100.0%100.0–100.0$0.0341—
QualityAutoBench60.824588.6%83.7–91.8$0.009766.3%
BalancedAutoBench60.324587.8%83.3–91.8$0.006659.9%
CostAutoBench59.824286.8%83.1–91.3$0.003778.9%
Always-LunaBaseline56.125185.3%80.9–89.6$0.0043—
Always-SolBaseline48.924887.5%83.5–91.5$0.0787—
ND k0Not Diamond48.924887.5%83.5–91.5$0.0787100.0%Yes
Azure balancedAzure Model Router47.017982.7%78.8–89.4$0.020448.0%
Azure qualityAzure Model Router47.023885.3%80.3–89.1$0.064740.9%
daridari39.324479.1%73.8–84.0$0.022014.7%
Auto high (Aug)OpenRouter39.224479.1%73.4–83.6$0.023057.9%
ND k4Not Diamond39.024980.3%75.5–85.2$0.060077.0%
Auto max (Aug)OpenRouter37.124078.3%72.9–83.3$0.028978.2%
ConiferConifer30.525074.8%69.2–80.0$0.005276.6%
Auto medium (Aug)OpenRouter26.724673.6%67.1–78.0$0.013178.2%
ND nativeNot Diamond21.325072.0%66.0–77.2$0.059763.9%
NadirClawNadirClaw16.924970.3%64.3–75.9$0.051458.7%
Orca qualityOrcaRouter3.524565.7%59.2–71.0$0.0005100.0%Yes
Auto low (Aug)OpenRouter3.524565.7%59.2–71.0$0.000698.4%Yes
Auto Router highOpenRouter1.624465.2%59.0–70.9$0.005325.0%
Always-cheapestBaseline0.025244.4%38.5–50.4$0.0002—
Orca balancedOrcaRouter0.025239.3%33.3–45.2$0.000299.6%Yes
Orca cheapestOrcaRouter0.025239.3%33.3–45.2$0.000299.2%Yes
ND k5Not Diamond0.025146.6%40.2–52.6$0.000982.9%
Auto Router mediumOpenRouter0.024757.9%51.8–64.0$0.002143.3%
Auto Router defaultOpenRouter0.024658.1%52.0–64.2$0.002439.3%
Auto Router lowOpenRouter0.024760.3%54.3–66.4$0.002538.5%
BrickBrick0.024163.9%57.7–69.7$0.003180.2%
Azure costAzure Model Router0.07445.9%33.8–56.8$0.009159.1%
CascadeCascade0.023560.9%53.2–66.0$0.010458.3%
RandomBaseline0.025264.7%58.7–70.6$0.01157.1%

ARI is the AutoBench Router Index at β = 0.2. The board is ordered by it, highest first. It runs from 0 to 100 and is higher when a pass is both accurate and cheap. A pass no better than random scores 0. How ARI is calculated. Accuracy is the share of questions where the chosen model is statistically as good as the best available answer.

Same-pick

Median cross-vendor same-pick is 0 across 59 pairs. The grid is the 12-pass comparison from the report. A cell is the share of questions on which the two passes named the same model.

Orca cheapOrca balOrca qualAzure costAzure balAzure qualNadirClawBrickCascadeND k5Auto highAuto low
Orca cheap1.000.990.010.140.010.000.020.000.010.000.000.01
Orca bal0.991.000.000.140.010.000.020.000.000.000.000.01
Orca qual0.010.001.000.000.000.000.000.800.580.000.000.98
Azure cost0.140.140.001.000.190.030.000.000.000.000.000.00
Azure bal0.010.010.000.191.000.110.010.000.000.000.000.01
Azure qual0.000.000.000.030.111.000.120.000.000.000.010.00
NadirClaw0.020.020.000.000.010.121.000.000.000.340.000.00
Brick0.000.000.800.000.000.000.001.000.480.000.000.79
Cascade0.010.000.580.000.000.000.000.481.000.000.000.57
ND k50.000.000.000.000.000.000.340.000.001.000.000.00
Auto high0.000.000.000.000.000.010.000.000.000.001.000.00
Auto low0.010.010.980.000.010.000.000.790.570.000.001.00

Scored on the answer matrix published as AutoBench Run 6.

Download the file

AutoBench Router Benchmark 2 - November 2026 - AutoBench