Delivering Transparency in LLM Benchmarking

AutoBench aims to overcome the limitations of traditional, static benchmarks.
The main goals are:

• To create a dynamic and “hard to hack” benchmark by generating new questions in each run.
• To reduce human bias and subjectivity in evaluations.
• To offer a highly scalable and cost-effective solution for frequent, large-scale LLM evaluation.
• To provide granular, all-domains capable insights into an LLM’s strengths and weaknesses.

AutoBench has demonstrated strong performance and potential:

• High Correlation with Established Benchmarks: It achieves over 85-95% correlation with generalist static benchmarks, such Artificial Analysis Intelligence Index (AAII), and ELO score human-based benchmarks, such as LMArena.
• Exceptional Cost-Efficiency: A full benchmark run evaluating over 30 leading LLMs, with over 20 LLM rankers, costs between $1,000 and $10,000 depending on the granularity (number of domains benchmarked) and the models used (‘Reasoning’ models tend to cost more).
• Dynamic and Robust: The system is difficult to “game” because it dynamically generates questions for each evaluation.

Unlike traditional benchmarks that use static datasets, AutoBench is dynamic. It generates new questions for every evaluation, which prevents models from being “trained to the test” or “gaming” the system. This approach tests for genuine general abilities rather than memorized answers.

AutoBench shifts the evaluation from subjective human bias to a transparent “LLM ecosystem bias”. The “Collective-LLM-as-a-Judge” approach mitigates the biases of any single model by aggregating the “collective view” of the entire LLM ecosystem. Its bias can be considered an average of the biases introduced into the ecosystem by the vast range of training datasets used to train models. The benchmark, therefore, measures performance relative to the consensus of contemporary AI systems.

Dynamic and Adaptive: Constantly evolves, making it hard to game.
Scalable and Cost-Effective: Can evaluate many models at a low cost (under $100 for 20 models).

Reduces Human Bias: Replaces subjective human evaluation with a collective model-based perspective.

Granular flexibility: it can be easily adapted to evaluate any chosen domain.
Stable and Reliable: The iterative weighting mechanism converges quickly, ensuring stable and reliable results.

“ASI ready”: it will keep working also when it will become challenging for humans to produce adequante benchmarking datasets and evaluations

AutoBench operates in a fully automated, iterative process:

• Dynamic Question Generation: In each iteration, a topic and difficulty are randomly selected, and one of the LLMs is tasked with generating a question.
• Question Quality Control: The generated question is then ranked by all other LLMs. Only questions that meet a specific quality threshold are accepted.
• Parallel Answer Generation: All LLMs generate answers to the accepted question in parallel.
• Parallel Answer Ranking: All LLMs then rank the answers provided by every other model on a 1-5 scale based on criteria like clarity, relevance, and correctness.
• Weighted Rank Aggregation: The individual ranks are aggregated into a weighted average for each answer. The weights of the “judge” LLMs are adapted over iterations based on their performance, giving more influence to models that provide consistently better judgments.

AutoBench evaluates LLM performance using two main types of metrics:

• General Average Rank: This is the primary metric for a model’s overall performance. It is calculated as the weighted average of all ranks a model receives across every question and topic in the benchmark run.
• Topic-Specific Average Ranks: To provide a more detailed view of a model’s capabilities, AutoBench calculates average ranks for a wide range of specific topics. These currently include: General Culture, Logics, Grammar, Science, Technology, Current News & History, Creative Writing, Math, Coding.

Beyond the quality of the answers, AutoBench also measures key performance indicators related to speed and cost:

• Average Answer Duration: The average time (in seconds) a model takes to generate a response.
• P99 Answer Duration: The 99th percentile of response time, which helps to understand a model’s worst-case latency.
• Average Answer Cost: The estimated cost to generate answers, typically measured in USD per million tokens.
• These granular scores help identify the specific strengths and weaknesses of each LLM.

Question quality is maintained through a rigorous, model-driven quality control process. A generated question is only accepted if its average quality rank from the judging LLMs is above 4.2  (on a 1-5 scale). If a question is rejected, the system automatically generates a new one.

Choosing the right LLM involves balancing three key factors: quality, speed, and cost. AutoBench is designed to help you navigate this trade-off.

• For Highest Quality: If your priority is the best possible answer, look at the General Average Rank.
• For Best Speed: If you need fast responses for a real-time application, focus on the Average Answer Duration. The “AutoBench Rank vs Answering Time” chart is a perfect tool for visualizing this, showing models that offer a good balance of speed and quality.
• For Lowest Cost: If budget is your main concern, refer to the Average Answer Cost metric to find models that are more economical to run at scale.

By considering these three dimensions, you can select a model that is perfectly suited to your specific use case and budget. If you need more use case specific evaluations, try Bot Scanner.

Yes, AutoBench is an open-source project. The methodology and code are shared to foster community collaboration and drive advancements in LLM benchmarking. We encourage community contributions to improve topic lists, prompts, and the overall system.

Where can I find more information and resources? You can find all our models, leaderboards, and demos on our Hugging Face pages:

• Main Profile: https://huggingface.co/AutoBench
• AutoBench 1.0 Model & Code: https://huggingface.co/AutoBench/AutoBench_1.0
• Live Leaderboard: https://huggingface.co/spaces/AutoBench/AutoBench
• Leaderboard Interactive Demo: https://huggingface.co/spaces/AutoBench/AutoBench_1.0_Demo

The project is open source, and we encourage developers and researchers to use it. The best place to start is the AutoBench 1.0 Model & Code page on Hugging Face. There, you will find the necessary code, configuration details, and instructions to run the benchmark on your own systems or with your own models.

AutoBench was developed by eZecute S.R.L., an Italian startup builder and advisor. The project also benefited from the collaboration of AI entrepreneurs and the support of leading AI companies, such as Translated, and accademic institutions, such as University of Rome “La Sapienza” DIAG.

Bot Scanner is a user-facing platform that was born from the insights gained while developing AutoBench. While AutoBench is a professional framework for benchmarking LLMs, Bot Scanner is an everyday tool that allows any user to query multiple LLMs simultaneously and use other AIs to rank the responses. It essentially packages the sophisticated evaluation principles of AutoBench into an accessible, user-friendly interface.

No. All user-generated data on Bot Scanner remains strictly personal and is not used for any benchmarking purposes, including for AutoBench.