AutoBench Run 6 - August 2026

Twenty models scored on 252 questions across ten subject domains. Score, cost, and latency are the per-answer committee rank, billed cost, and generation time. Routers in the AutoBench Router Benchmark are scored against this answer matrix.

Latest

Date

August 29, 2026

Version

2026-08-29

Models

New Models

Run data

Overall Comparison Costs Latency P99 Latency Domain

P99 latency breakdown across 10 specialized domains, measured in seconds. This 99th percentile metric reveals worst-case response times, helping you ensure consistent performance for production environments.

Model	Average (All Topics)	Computer Science	Philosophy	Social Science	Language	Science	Technology	Arts	Literature	History	Agentic
Muse Glimmer 30B	82.39s (#1)	79.36s (#1)	47.82s (#4)	27.41s (#1)	51.06s (#2)	84.48s (#1)	59.76s (#3)	37.39s (#2)	24.91s (#1)	28.99s (#2)	15.21s (#2)
Gemini 3.7 Flash	116.93s (#3)	112.00s (#3)	44.49s (#3)	73.25s (#5)	103.81s (#6)	104.53s (#2)	106.85s (#6)	175.84s (#7)	46.79s (#5)	41.89s (#3)	19.77s (#4)
Gemma 4 31B IT	117.31s (#4)	150.80s (#6)	19.39s (#1)	72.74s (#4)	38.92s (#1)	117.35s (#3)	43.92s (#2)	128.34s (#6)	71.17s (#7)	54.03s (#7)	11.05s (#1)
Hy3	112.41s (#2)	90.68s (#2)	72.44s (#9)	56.28s (#3)	85.18s (#5)	127.40s (#4)	102.84s (#5)	70.58s (#4)	33.56s (#3)	42.28s (#4)	25.84s (#7)
Gpt oss 120b	145.77s (#6)	119.32s (#4)	65.65s (#8)	121.16s (#7)	82.84s (#4)	148.78s (#5)	219.60s (#12)	197.87s (#8)	110.27s (#13)	68.68s (#9)	22.48s (#6)
Grok 4.3	142.98s (#5)	139.08s (#5)	21.79s (#2)	32.97s (#2)	77.17s (#3)	190.17s (#6)	63.61s (#4)	34.02s (#1)	32.05s (#2)	21.19s (#1)	18.08s (#3)
GPT-5.6 Luna	298.19s (#10)	350.95s (#12)	160.19s (#17)	305.56s (#18)	244.04s (#11)	232.04s (#7)	259.32s (#13)	252.52s (#11)	212.75s (#17)	142.50s (#17)	70.10s (#15)
Nemotron 3 Ultra	322.09s (#11)	322.53s (#9)	148.41s (#14)	130.90s (#10)	354.53s (#14)	255.64s (#8)	453.09s (#19)	273.67s (#13)	60.63s (#6)	45.38s (#5)	97.26s (#17)
Muse Spark 1.2	193.84s (#7)	328.24s (#10)	102.80s (#11)	124.08s (#8)	117.26s (#7)	256.56s (#9)	172.00s (#10)	58.23s (#3)	103.19s (#11)	114.10s (#13)	68.67s (#14)
Kimi K2.6	487.51s (#19)	514.94s (#18)	185.84s (#18)	210.49s (#15)	559.35s (#20)	268.77s (#10)	378.15s (#18)	361.90s (#17)	167.32s (#16)	153.11s (#18)	171.78s (#19)
GLM 5.3	277.53s (#9)	482.28s (#16)	65.53s (#7)	264.70s (#16)	186.98s (#9)	273.80s (#11)	40.75s (#1)	271.20s (#12)	226.03s (#18)	76.93s (#11)	26.59s (#8)
Qwen3.8 Max	408.55s (#17)	256.71s (#7)	147.18s (#13)	158.09s (#13)	404.50s (#17)	300.07s (#12)	269.86s (#15)	452.31s (#19)	156.35s (#15)	123.57s (#14)	26.68s (#9)
MiniMax M3	376.61s (#14)	388.77s (#14)	154.50s (#15)	125.17s (#9)	231.23s (#10)	333.05s (#13)	183.63s (#11)	422.19s (#18)	110.44s (#14)	140.82s (#16)	32.41s (#10)
DeepSeek V4 Flash 0731	371.82s (#13)	485.43s (#17)	216.75s (#19)	273.77s (#17)	328.31s (#13)	339.48s (#14)	268.12s (#14)	302.65s (#15)	103.09s (#10)	75.03s (#10)	54.44s (#12)
GPT-5.6 Sol	400.97s (#16)	427.59s (#15)	158.64s (#16)	446.85s (#20)	369.60s (#16)	364.33s (#15)	270.72s (#16)	277.75s (#14)	340.12s (#20)	160.89s (#19)	58.05s (#13)
Claude Sonnet 5	381.49s (#15)	381.29s (#13)	104.85s (#12)	150.73s (#11)	360.23s (#15)	382.06s (#16)	140.66s (#8)	95.99s (#5)	92.07s (#9)	60.93s (#8)	21.86s (#5)
Nemotron 3.5 Lightning	343.88s (#12)	335.60s (#11)	47.99s (#5)	118.81s (#6)	279.25s (#12)	386.92s (#17)	304.95s (#17)	224.54s (#10)	37.27s (#4)	124.86s (#15)	136.10s (#18)
Grok 4.6	237.80s (#8)	516.02s (#19)	63.22s (#6)	173.83s (#14)	167.89s (#8)	394.91s (#18)	124.42s (#7)	216.79s (#9)	103.35s (#12)	86.53s (#12)	44.64s (#11)
GLM 5.2	472.50s (#18)	556.74s (#20)	87.88s (#10)	156.53s (#12)	464.45s (#18)	405.37s (#19)	168.32s (#9)	488.91s (#20)	73.26s (#8)	51.12s (#6)	71.60s (#16)
Qwen3.5-9B	488.25s (#20)	285.18s (#8)	262.10s (#20)	332.57s (#19)	509.58s (#19)	555.65s (#20)	469.04s (#20)	355.05s (#16)	328.50s (#19)	362.34s (#20)	317.22s (#20)