AutoBench Run 6 - August 2026

Twenty models scored on 252 questions across ten subject domains. Score, cost, and latency are the per-answer committee rank, billed cost, and generation time. Routers in the AutoBench Router Benchmark are scored against this answer matrix.

Latest

Date

August 29, 2026

Version

2026-08-29

Models

New Models

Run data

Overall Comparison Costs Latency P99 Latency Domain

AutoBench provides a complete dashboard with highly customizable domain-specific average ranks and precise efficiency metrics (average answer cost, average answer duration, P99 Latency). For further detail also check our Hugging Face Leaderboard.

Model	Score	Avg Cost ($ Cents)	Avg Latency (sec)	P99 Latency (sec)	Iterations
Qwen3.5-9B	4.15 (#19)	0.10 (#5)	144s (#20)	488s (#20)	223
Kimi K2.6	4.69 (#5)	1.45 (#16)	93s (#19)	488s (#19)	238
GPT-5.6 Sol	4.78 (#1)	7.87 (#20)	66s (#18)	401s (#16)	248
GPT-5.6 Luna	4.74 (#3)	0.43 (#8)	57s (#17)	298s (#10)	251
Qwen3.8 Max	4.74 (#2)	1.63 (#17)	55s (#16)	409s (#17)	246
DeepSeek V4 Flash 0731	4.59 (#12)	0.05 (#3)	53s (#15)	372s (#13)	245
GLM 5.2	4.67 (#7)	0.51 (#11)	52s (#14)	473s (#18)	247
Grok 4.6	4.60 (#11)	1.64 (#18)	48s (#13)	238s (#8)	247
Nemotron 3.5 Lightning	4.36 (#16)	0.15 (#7)	48s (#12)	344s (#12)	252
Claude Sonnet 5	4.73 (#4)	4.05 (#19)	45s (#11)	381s (#15)	243
MiniMax M3	4.65 (#8)	0.44 (#9)	44s (#10)	377s (#14)	244
GLM 5.3	4.65 (#9)	0.57 (#12)	38s (#9)	278s (#9)	243
Nemotron 3 Ultra	4.49 (#14)	0.69 (#13)	38s (#8)	322s (#11)	244
Muse Spark 1.2	4.56 (#13)	1.15 (#15)	33s (#7)	194s (#7)	251
Gpt oss 120b	4.12 (#20)	0.02 (#1)	32s (#6)	146s (#6)	252
Hy3	4.61 (#10)	0.09 (#4)	27s (#5)	112s (#2)	252
Gemini 3.7 Flash	4.68 (#6)	1.07 (#14)	22s (#4)	117s (#3)	251
Grok 4.3	4.44 (#15)	0.50 (#10)	19s (#3)	143s (#5)	252
Muse Glimmer 30B	4.35 (#17)	0.11 (#6)	17s (#2)	82s (#1)	252
Gemma 4 31B IT	4.35 (#18)	0.02 (#2)	16s (#1)	117s (#4)	252