Back to Blog

Try an LLM Router That Actually Routes ($1 free credit)

First we benchmarked the available routers: they all failed. Then we built a better one on the same eval engine. Three settings, three models, and $1 of free credit to try it.

AutoBench Team•October 29, 2026•10 min read
RouterBeta Router

First we benchmarked the available routers: They all failed. Then we realized we could build a better one based on the same eval engine. The Autobench Router: three settings, three models, no collapse onto one model, and the top score on the Router Index. $1 of free credit to try it.

Our Router scores better than the best model in our pool. The cost? Only 8% of the price of that same model. We built it by training an AI model on synthetic data generated by the same eval engine that gave us the router benchmark.

The Beta Router is live on our user-facing service: botscanner.ai. Try the chat at botscanner.ai/router, or point an existing agent at the OpenAI-compatible API. New accounts get $1 of free credit — about 270 hard prompts on the Cost setting at billed model cost (Existing Botscanner users also get the $1 extra credits - check your inbox). For the board it was measured against, see our previous post, Do LLM Routers Actually Route?.

Best Answer Rate (Delta-Hit) against billed answer cost. Commercial routers are grey. The three Beta Router knobs sit as green stars inside the prize — above random quality and cheaper than a random pick.

Figure 1. The Beta Router vs seven commercial and open-source routers (17 configurations). The Beta Router is the only one that consistently stays in the prize area — above random quality (64.7%) and left of random cost ($0.0115). At Balanced and Quality, it returns higher quality than the top model (GPT-5.6 Sol) at respectively 8% and 12% of the cost. At Cost, it scores higher and cheaper than a mid-cost model like GPT-5.6 Luna. All other routers don't even come close.

The Beta Router currently operates on three models (low, mid, high): DeepSeek V4 Flash, GPT-5.6 Luna, GPT-5.6 Sol (we are already working to expand and update the models available). We tested it on the same router benchmark we released last Monday.

Key findings

  • Balanced already beats Sol. 87.8% Delta-Hit at $0.0066 per question, against always calling GPT-5.6 Sol at 87.5% and $0.0787. That is the 8% bill in the hook.
  • Quality is a further lift. 88.6% at $0.0097 — 1.1 points above Sol, still 88% cheaper.
  • Cost beats Luna on both axes. 86.8% at $0.0037, against always calling GPT-5.6 Luna at 85.3% and $0.0043. It does not beat Sol on quality, and we do not claim it does.
  • All three knobs actually route. None sends 90% of prompts to one model. Quality, Balanced, and Cost spread their picks across DeepSeek, Luna, and — on the first two knobs — Sol.
  • Highest Router Index on the board. At the official cost weight, Quality scores 60.8, Balanced 60.3, and Cost 59.8. Always-Luna, the best static policy on Monday's index, is 56.1.
  • It is live, with $1 to try it. Chat, or drop the API into an existing agent. A $1 free credit is about 270 hard prompts on the Cost setting at billed model cost — against about 13 if you always called Gpt 5.6 Sol.

Back to the prize

In our Router Benchmark (released last Monday), we showed that, in order to generate real routing value, a router needs to sit in the region above random quality and cheaper than a random pick (64.7% at $0.0115, see the Pareto graph in Fig. 1 above). Always calling GPT-5.6 Luna already sits there as a static policy: 85.3% at $0.0043. Always calling GPT-5.6 Sol, the best single model in the 20-model pool, scores 87.5% at $0.0787. A router earns its keep only if it enters that region — and, to be worth paying for, if it also beats those statics.

Seventeen headline settings from seven commercial and open systems left the region empty. The cheap settings sat on the random line because they had collapsed onto one flash model. The strong settings bought a strong model and cost more than random. No pass was both. In 10,000 bootstrap resamples, none ever beat Luna on quality and cost together.

Figure 1 is that chart again, with the commercial points greyed out. The three green stars are the Beta Router.

What we built

Same eval engine, new data, new router

The AutoBench Router Benchmark grades a router by lookup: the router names a model, and we read that model's already-judged, already-priced answer. We used that same method (not the same questions) to build a separate training set of synthetic data. For each training prompt we stored DeepSeek, Luna, and Sol answers with committee scores and billed costs, then trained an AI model on those synthetic, judge-scored triples.

Three knobs, one dial

You pick Quality, Balanced, or Cost. That is the only product control. Quality and Balanced may name Sol. Cost names DeepSeek or Luna only. You cannot pin a model yourself.

What we found

Balanced already beats Sol

The hook is the Balanced knob. 87.8% against Sol's 87.5%, at $0.0066 against $0.0787 — about a twelfth of the bill, 8% of Sol's price. Quality sits further up: 88.6% at $0.0097, still 88% cheaper than always-Sol.

Zoom of Always-Sol, Always-Luna, and the three Beta Router knobs.

Figure 2. The prize, zoomed to 84–90% Best Answer rate (Delta-Hit). Balanced is already above Always-Sol at about a twelfth of the bill. Quality is a further lift. Cost sits above and left of Always-Luna.

As worthy as Luna, and cheaper

Always-Luna was the main bar that no commercial or open-source router could beat in our benchmark. Luna offers a good compromise on quality (just a few % points below Sol) at about 5% of the bill, and no router required. The Cost knob clears that bar on both axes: 86.8% at $0.0037, against 85.3% at $0.0043. It is 95% cheaper than always-Sol. It is not a Sol-beating setting, and Figure 2 does not pretend it is.

It actually routes

The main question we asked in our router benchmark was whether routers actually route. Five of the seventeen commercial settings sent at least 90% of prompts to one model, so their quality number was that model's number. A quality knob often just changed which constant you got.

The Beta Router does not do that. On the benchmark, Quality sends 42 prompts to DeepSeek, 167 to Luna, and 33 to Sol. Balanced sends 63 / 151 / 28. Cost sends 51 / 191 / 0 (DeepSeek and Luna only, by design). The largest single-model share is 69%, 62%, and 79%. All three sit under the 90% collapse line.

Stacked dispatch mix for the three knobs, next to five collapsed Wave passes.

Figure 3. Share of decisions. Green labels are Beta Router knobs, stacked by DeepSeek / Luna / Sol, with the largest-model share on the left. Grey bars are the five Wave settings that collapse onto one model. The dashed line is 90%.

Top of the Index

Best-Answer rate (also defined as Delta-Hit) and dollars sit on different axes. One measures response quality, the other cost. The Autobench Router Index (ARI) answers the joint question at a locked cost weight β = 0.2. Quality is how far Delta-Hit sits above the frozen random policy. Cost is a log score between two fixed dollar anchors. A pass that does not beat random on quality scores ARI 0.

In our router benchmark, Always-Luna led at 56.1. No commercial or open-source router had positive lift against it. Ours does. With the three knobs on the same index (using the Wave random floor as an approximation) Quality is 60.8, Balanced 60.3, and Cost 59.8. All three sit above Always-Luna.

Autobench Router Index bars at official β = 0.2, with the three knobs added above Always-Luna.

Figure 4. Official Autobench Router Index at β = 0.2. Green bars are the Beta Router knobs. Black bars are Always-Luna and Always-Sol. Blue bars are commercial and open-source routers above zero. The knobs use the Wave random floor as an approximation; they were scored on the benchmark dataset.

Knob by knob

  • Quality (autobench/beta-quality) is the accuracy lift: 88.6% at $0.0097, 88% cheaper than always-Sol and 1.1 points above it. It still names Sol on 33 of 242 prompts. Use it when the last point of quality is worth a fraction of a cent.
  • Balanced (autobench/beta-balanced) is the hero setting: 87.8% at $0.0066. It already beats always-Sol on quality, at 8% of Sol's bill. This is the default we show in the snippet below.
  • Cost (autobench/beta-cost) is the cheaper Luna-beating point: 86.8% at $0.0037. It never names Sol. It is 95% cheaper than always-Sol and beats Always-Luna on both axes. A $1 credit is about 270 hard prompts at this setting at billed model cost. Do not read it as a Sol-level setting.

The fine print

These numbers are a beta, on a frozen holdout, with a three-model pool.

  • The training set and the benchmark board are different question sets. They share a method (the same judges, the same prices, the same Delta-Hit rule) not the same prompts. The board is a holdout from the question corpus. It is not a silent restamp of the scoreboard, and it is not a slice of the training data. Figure 1 puts both sets of points on one plane so you can see the prize.
  • The router chooses among three models — DeepSeek, GPT-5.6 Luna, and GPT-5.6 Sol — not among the twenty in the benchmark pool. Cost chooses among two.
  • Cost does not beat always-Sol on quality. Only Quality and Balanced do.
  • Published quality is single-turn. We are planning the training of a router that can handle also multi-turn agentic loads.

Try it

The Beta Router is live now, as a chat page and as an OpenAI-compatible API you can drop into an existing agent. New accounts get $1 of free credit — about 270 hard prompts on the Cost setting at billed model cost. Existing Botscanner users: check your inbox.

from openai import OpenAI

client = OpenAI(base_url="https://api.autobench.org/v1", api_key="YOUR_KEY")
out = client.chat.completions.create(
    model="autobench/beta-balanced",
    messages=[{"role": "user", "content": "…"}],
)

autobench/beta-quality and autobench/beta-cost are the other two knobs. Pay-as-you-go is 1.5× the billed OpenRouter cost of the model the router named.

A $1 free credit buys about 270 Cost prompts at billed model cost — against about 13 if you always called Sol, and about 150 on Balanced. Figure 5 is that comparison. Pay-as-you-go customer price is 1.5× the billed bars, so a spent dollar of wallet buys fewer prompts than these counts.

Prompts per billed dollar for Always-Sol, Always-Luna, and the three knobs.

Figure 5. Prompts per $1 at billed OpenRouter cost. Always-Sol buys about 13. Balanced buys about 150. Cost buys about 270. Pay-as-you-go customer price is 1.5× these costs.

What's next

We plan to release versions that can handle more recent models, and to extend the training set for accuracy and use-case coverage. Published quality is single-turn. We plan a version trained on multi-turn agentic usage as well.

If you run a router you want on the board, or a workload you want a custom router trained on, send it.

Join the Community

The technical report is Autobench's white paper Do LLM Routers Pay for Themselves?