AutoBench Run 6 - August 2026

Twenty models scored on 252 questions across ten subject domains. Score, cost, and latency are the per-answer committee rank, billed cost, and generation time. Routers in the AutoBench Router Benchmark are scored against this answer matrix.

Latest

Date

August 29, 2026

Version

2026-08-29

Models

New Models

Run data

Overall Comparison Costs Latency P99 Latency Domain

Average latency breakdown across 10 specialized domains, measured in seconds. These metrics help you identify models that consistently deliver quick responses for your specific use case while maintaining quality standards.

Model	Average (All Topics)	Computer Science	Philosophy	Social Science	Language	Science	Technology	Arts	Literature	History	Agentic
Qwen3.5-9B	144.22s (#20)	131.56s (#18)	138.80s (#20)	131.91s (#20)	202.57s (#20)	183.40s (#20)	176.48s (#20)	149.72s (#20)	137.27s (#20)	122.29s (#20)	92.40s (#20)
Kimi K2.6	93.06s (#19)	199.94s (#20)	75.55s (#19)	78.23s (#19)	126.02s (#19)	78.12s (#15)	144.21s (#19)	101.77s (#19)	64.07s (#19)	57.07s (#19)	46.17s (#19)
GLM 5.2	52.38s (#14)	115.30s (#16)	33.16s (#12)	34.78s (#9)	90.25s (#18)	92.81s (#19)	51.13s (#12)	57.43s (#13)	21.51s (#8)	18.61s (#7)	11.39s (#11)
GPT-5.6 Sol	66.32s (#18)	146.26s (#19)	36.84s (#13)	75.42s (#18)	79.54s (#17)	75.50s (#14)	89.32s (#18)	75.40s (#17)	55.66s (#18)	39.16s (#18)	10.18s (#8)
Qwen3.8 Max	54.83s (#16)	87.51s (#10)	50.23s (#16)	51.38s (#15)	74.81s (#16)	63.71s (#12)	65.68s (#16)	85.49s (#18)	39.19s (#14)	30.41s (#14)	14.13s (#15)
DeepSeek V4 Flash 0731	53.47s (#15)	124.68s (#17)	51.88s (#18)	49.07s (#14)	64.52s (#15)	82.12s (#16)	58.67s (#13)	52.95s (#12)	29.77s (#9)	26.00s (#12)	12.05s (#13)
Nemotron 3.5 Lightning	48.26s (#12)	109.47s (#14)	17.15s (#5)	33.18s (#8)	57.96s (#14)	88.26s (#18)	61.40s (#15)	36.66s (#8)	19.39s (#7)	22.65s (#10)	27.60s (#18)
GPT-5.6 Luna	56.52s (#17)	110.46s (#15)	48.75s (#15)	73.73s (#17)	55.36s (#13)	55.20s (#11)	87.72s (#17)	62.84s (#15)	52.06s (#17)	30.99s (#15)	16.39s (#16)
Nemotron 3 Ultra	38.07s (#8)	85.53s (#9)	32.16s (#11)	32.24s (#7)	54.21s (#12)	39.80s (#5)	58.82s (#14)	44.18s (#11)	18.58s (#6)	17.26s (#6)	11.54s (#12)
Claude Sonnet 5	45.07s (#11)	104.13s (#12)	40.08s (#14)	44.04s (#12)	52.21s (#11)	84.02s (#17)	45.25s (#9)	29.84s (#7)	30.73s (#11)	21.52s (#9)	9.20s (#4)
Grok 4.6	48.43s (#13)	105.93s (#13)	30.98s (#10)	45.86s (#13)	49.83s (#10)	65.90s (#13)	46.95s (#10)	60.12s (#14)	39.80s (#15)	34.72s (#17)	12.96s (#14)
MiniMax M3	44.20s (#10)	93.03s (#11)	50.94s (#17)	42.12s (#10)	40.56s (#9)	44.14s (#8)	48.83s (#11)	70.62s (#16)	33.08s (#12)	32.76s (#16)	10.99s (#10)
GLM 5.3	38.26s (#9)	78.66s (#8)	25.37s (#7)	55.66s (#16)	36.62s (#8)	51.44s (#10)	22.60s (#4)	43.49s (#10)	44.15s (#16)	22.74s (#11)	10.11s (#7)
Gpt oss 120b	32.29s (#6)	47.02s (#6)	29.06s (#8)	42.45s (#11)	31.53s (#7)	40.14s (#6)	43.00s (#8)	38.71s (#9)	29.99s (#10)	21.01s (#8)	9.72s (#6)
Muse Spark 1.2	33.32s (#7)	54.69s (#7)	30.81s (#9)	27.26s (#6)	29.87s (#6)	50.00s (#9)	40.63s (#7)	23.53s (#4)	37.73s (#13)	29.77s (#13)	17.40s (#17)
Hy3	26.58s (#5)	46.53s (#5)	24.12s (#6)	25.19s (#5)	27.94s (#5)	41.42s (#7)	32.97s (#6)	27.54s (#5)	15.57s (#4)	16.91s (#5)	10.65s (#9)
Grok 4.3	19.37s (#3)	36.26s (#3)	12.26s (#2)	14.99s (#1)	21.19s (#4)	38.89s (#4)	19.93s (#3)	14.72s (#1)	14.10s (#2)	10.63s (#1)	9.25s (#5)
Gemini 3.7 Flash	21.56s (#4)	40.61s (#4)	16.87s (#4)	20.71s (#4)	21.16s (#3)	27.47s (#2)	27.44s (#5)	27.79s (#6)	16.56s (#5)	15.89s (#4)	6.09s (#2)
Muse Glimmer 30B	17.02s (#2)	23.97s (#1)	15.27s (#3)	15.83s (#2)	19.18s (#2)	28.57s (#3)	19.91s (#2)	15.65s (#2)	11.57s (#1)	11.51s (#2)	8.75s (#3)
Gemma 4 31B IT	15.89s (#1)	29.76s (#2)	8.75s (#1)	17.08s (#3)	13.16s (#1)	27.07s (#1)	18.45s (#1)	17.17s (#3)	14.35s (#3)	11.54s (#3)	3.13s (#1)