PK
Peter Kruger
CEO
Serial technology entrepreneur, investor and executive. M.Sc. Physics, specialising in the statistical mechanics of neural networks. 30+ startup investments including Robot.com, Serket and EVJA.
Evidence,NotGuesswork
AutoBench measures models and routers, and routes each prompt to a model that is good enough at the lowest cost.
Our mission
Measure models and routers, and send each prompt to a model that is good enough at the lowest cost. The engine writes the questions, scores the answers, and keeps the evidence that both a benchmark and a routing decision are graded on.
Why AutoBench exists
Static leaderboards get gamed. Human reviews are slow, subjective, and expensive. New models land every week. Routing without a measurement of which model was good enough is a guess. AutoBench writes fresh questions, lets models judge each other, and keeps the record.
One engine: benchmarks and routing evidence
The same run that ranks models is the trace a router is graded on. Questions are new each run. Answers carry their cost and their latency. A committee of models scores them.
- 01
Generate
Fresh, difficulty-balanced questions. No fixed public test set.
- 02
Answer
Every model answers. Cost and latency stay attached to the answer.
- 03
Judge
A committee of models scores every answer. Consensus is the signal.
- 04
Use the evidence
Publish a benchmark, or grade a router against the same matrix.
We don't build base models; we measure them. Neutrality is the product.
What makes us different
Collective-LLM-as-a-Judge
Models judge each other to dilute single-model bias and reflect the state of the ecosystem.
Dynamic by design
New prompts every run resist benchmark gaming and stay relevant as capabilities shift.
Proven alignment
The latest public model run correlates 88.13% with the Artificial Analysis Intelligence Index. DIAG at Sapienza University independently validated the methodology (arXiv:2510.22593).
One record, two uses
A finished run is a model benchmark and the matrix a router is scored against. Routing evidence does not need a second methodology.
How we work with you
Managed benchmark runs
We operate the benchmark for you, private to your data and endpoints, and deliver a decision-ready report.
Router benchmark and custom router
Grade a router on your traffic, or have a router built for your model pool from that evidence.
In-cloud deployment
Prefer to run it yourself? Set the engine up in your environment and plug in your model list.
The founding team
OP
Oleksandr Pryymak
CSO
Former Meta AI scientist, PhD. Brings scientific leadership and operational excellence. Expert in LLM evaluation. A key contributor to the platform's core software development and technical validation.
CG
Costanza Gallo
COO
Builds and manages customer and partner relationships. Enterprise SaaS sales at Oracle, ventures in Web3. M.Sc. London School of Economics.
Benchmark models. Grade routers.
Use the public leaderboards, read the Router Benchmark, or ask us to run the engine on your models and your traffic. The data is also on Hugging Face.
Our Partners
DIAG independently validated the methodology (arXiv:2510.22593).