First we benchmarked the available routers: they all failed. Then we built a better one on the same eval engine. Three settings, three models, and $1 of free credit to try it.
Introducing the AutoBench Router Benchmark, a pre-registered, cost-aware audit of major LLM routers.
Based on LLM generated virtual agents, it handles countless agentic tasks to offer unbiased and granular LLM evaluation.
We are also announcing our latest benchmark (Run 5) made possible by new platform features for more powerful and efficient benchmarking: Random Score Pooling, Nonlinear Weighting, Parallel Iteration.
We teamed up with leading agritech company EVJA to drop the first-ever LLM benchmark dedicated to the agricultural sector. 40 models, 4 professional personas, and one major open-source surprise.
This run evaluated 33 models across over 300 iterations (generated questions) using 21 ranking models and generating over 220,000 individual rankings.
We're thrilled to announce that AutoBench has moved from a promising open-source project to a scientifically validated framework, with our first paper published in collaboration with Sapienza University of Rome.