AutoBench Run 5 - December 2025

Latest AutoBench run with models Gpt 5.2, Claude Opus 4.5, DeepSeek 3.2 Speciale and more

Latest

Date

December 16, 2025

Version

2025-12-16

Models

New Models

Run data

Overall Comparison Costs Latency P99 Latency Domain

P99 latency breakdown across 10 specialized domains, measured in seconds. This 99th percentile metric reveals worst-case response times, helping you ensure consistent performance for production environments.

Model	Average (All Topics)	Coding	Creative Writing	Current News	General Culture	Grammar	History	Logics	Math	Science	Technology
grok-4.1-fast	57.44s (#1)	49.75s (#1)	20.18s (#2)	54.43s (#1)	78.25s (#3)	46.27s (#2)	49.49s (#1)	77.61s (#1)	71.87s (#3)	48.17s (#1)	78.36s (#2)
gemini-2.5-flash-lite	69.09s (#2)	78.70s (#2)	11.68s (#1)	128.71s (#4)	38.80s (#1)	44.97s (#1)	110.24s (#8)	86.64s (#3)	74.84s (#4)	84.66s (#2)	31.65s (#1)
Nova-2-lite-v1	131.60s (#3)	173.45s (#5)	103.96s (#15)	157.20s (#5)	159.48s (#11)	133.87s (#5)	108.75s (#7)	116.05s (#5)	119.65s (#6)	135.61s (#6)	107.99s (#4)
nova-premier-v1	134.54s (#4)	177.99s (#6)	35.03s (#4)	225.56s (#12)	188.51s (#13)	93.04s (#4)	103.82s (#5)	83.18s (#2)	94.83s (#5)	145.68s (#7)	197.75s (#12)
Mistral-medium-3.1	146.93s (#5)	101.37s (#3)	25.43s (#3)	207.95s (#10)	212.72s (#16)	182.36s (#12)	175.82s (#14)	108.12s (#4)	63.58s (#1)	166.40s (#9)	225.52s (#19)
Ministral-8b-2512	154.14s (#6)	108.05s (#4)	233.48s (#26)	256.42s (#16)	82.40s (#4)	55.00s (#3)	104.63s (#6)	423.65s (#15)	68.72s (#2)	92.10s (#3)	116.93s (#5)
gemini-2.5-flash	173.94s (#7)	475.33s (#18)	58.33s (#6)	184.80s (#8)	139.99s (#8)	178.27s (#10)	125.50s (#11)	154.38s (#7)	178.77s (#8)	112.88s (#4)	131.14s (#6)
Gpt-oss-20b	183.02s (#8)	279.10s (#7)	37.39s (#5)	221.63s (#11)	94.44s (#5)	235.04s (#14)	117.91s (#9)	253.53s (#10)	225.41s (#11)	147.66s (#8)	218.10s (#17)
gemini-3-pro-preview	186.15s (#9)	381.70s (#15)	98.86s (#12)	168.30s (#7)	140.84s (#9)	157.04s (#7)	98.37s (#3)	342.94s (#13)	247.19s (#12)	129.19s (#5)	97.11s (#3)
Mistral-large-2512	198.13s (#10)	318.52s (#13)	67.47s (#8)	277.47s (#18)	230.39s (#18)	182.53s (#13)	212.87s (#20)	132.59s (#6)	141.92s (#7)	204.26s (#12)	213.26s (#16)
grok-4.1-fast-thinking	207.02s (#11)	285.18s (#9)	73.44s (#9)	100.46s (#2)	138.18s (#7)	177.54s (#9)	99.14s (#4)	516.52s (#19)	266.01s (#14)	246.66s (#14)	167.10s (#10)
nemotron-nano-9b-v2	211.73s (#12)	338.66s (#14)	195.02s (#22)	111.18s (#3)	64.85s (#2)	178.36s (#11)	92.15s (#2)	473.75s (#16)	272.36s (#16)	169.01s (#10)	222.00s (#18)
gemini-2.5-pro	222.47s (#13)	306.23s (#11)	63.38s (#7)	314.20s (#22)	209.90s (#14)	160.49s (#8)	139.17s (#12)	328.10s (#12)	268.02s (#15)	228.24s (#13)	207.00s (#14)
Qwen3-next-80b-a3b-thinking	226.68s (#14)	497.62s (#19)	107.44s (#16)	187.07s (#9)	154.37s (#10)	247.60s (#16)	120.27s (#10)	189.55s (#8)	343.78s (#17)	279.02s (#21)	140.04s (#7)
llama-3.3-nemotron-super-49b-v1.5	240.32s (#15)	408.06s (#17)	271.44s (#29)	160.54s (#6)	133.01s (#6)	155.29s (#6)	142.28s (#13)	299.67s (#11)	423.42s (#20)	257.52s (#16)	151.94s (#8)
gpt-5-mini	257.83s (#16)	560.12s (#21)	166.98s (#18)	239.95s (#15)	185.68s (#12)	267.72s (#19)	200.70s (#18)	246.31s (#9)	214.93s (#10)	269.22s (#18)	226.66s (#20)
gpt-5-nano	268.83s (#17)	282.18s (#8)	98.87s (#13)	238.91s (#14)	231.81s (#19)	302.92s (#22)	185.82s (#15)	490.26s (#17)	440.47s (#22)	259.70s (#17)	157.38s (#9)
gpt-oss-120b	291.84s (#18)	309.68s (#12)	98.59s (#11)	308.84s (#21)	258.63s (#21)	313.48s (#23)	194.04s (#17)	684.89s (#26)	180.43s (#9)	380.11s (#28)	189.75s (#11)
claude-haiku-4.5	316.92s (#19)	303.69s (#10)	172.93s (#19)	283.97s (#19)	514.83s (#30)	246.10s (#15)	325.97s (#26)	418.19s (#14)	361.13s (#19)	252.93s (#15)	289.47s (#24)
Kimi-K2-0905	329.01s (#20)	537.40s (#20)	227.50s (#25)	434.36s (#29)	300.18s (#23)	262.83s (#18)	233.52s (#22)	537.19s (#22)	355.69s (#18)	188.43s (#11)	212.96s (#15)
Qwen3-235B-A22B-2507	337.21s (#21)	388.67s (#16)	256.53s (#28)	274.04s (#17)	224.98s (#17)	386.17s (#25)	290.78s (#25)	524.49s (#20)	507.05s (#23)	284.57s (#24)	234.87s (#21)
Claude-opus-4.5	373.31s (#22)	834.12s (#24)	210.10s (#23)	329.94s (#23)	377.65s (#26)	254.76s (#17)	382.09s (#27)	493.52s (#18)	264.77s (#13)	281.70s (#22)	304.43s (#25)
Deepseek-v3.2	410.46s (#23)	631.21s (#22)	193.29s (#21)	356.02s (#25)	313.10s (#25)	393.95s (#26)	395.74s (#28)	541.90s (#23)	691.56s (#25)	278.37s (#20)	309.50s (#26)
GLM-4.5-Air	425.28s (#24)	945.31s (#27)	88.57s (#10)	379.63s (#26)	236.09s (#20)	270.87s (#20)	253.79s (#24)	632.83s (#25)	900.50s (#27)	274.40s (#19)	270.86s (#23)
Gpt-5.2	434.15s (#25)	983.23s (#28)	173.15s (#20)	458.68s (#30)	401.14s (#27)	278.51s (#21)	244.60s (#23)	743.59s (#28)	574.40s (#24)	283.53s (#23)	200.63s (#13)
Minimax-m2	472.81s (#26)	991.82s (#30)	589.49s (#33)	238.88s (#13)	288.48s (#22)	400.58s (#27)	201.58s (#19)	562.56s (#24)	746.35s (#26)	374.69s (#26)	333.69s (#28)
deepSeek-R1-0528	476.65s (#27)	844.76s (#25)	99.46s (#14)	344.66s (#24)	310.40s (#24)	362.74s (#24)	220.56s (#21)	737.76s (#27)	1043.45s (#28)	548.97s (#33)	253.75s (#22)
claude-sonnet-4.5	476.75s (#28)	847.62s (#26)	216.36s (#24)	502.25s (#32)	555.51s (#32)	406.21s (#28)	564.10s (#33)	526.61s (#21)	428.48s (#21)	340.18s (#25)	380.19s (#30)
grok-4	562.46s (#29)	1010.63s (#31)	281.41s (#30)	304.53s (#20)	211.69s (#15)	448.43s (#29)	189.40s (#16)	1420.30s (#32)	1052.80s (#29)	377.39s (#27)	327.97s (#27)
gpt-5.1	627.33s (#30)	991.04s (#29)	369.43s (#31)	416.66s (#28)	430.18s (#28)	560.75s (#33)	468.29s (#32)	1079.65s (#30)	1077.59s (#30)	512.74s (#30)	366.99s (#29)
GLM-4.6	630.49s (#31)	726.63s (#23)	152.22s (#17)	537.89s (#35)	591.02s (#35)	502.27s (#30)	618.98s (#35)	898.32s (#29)	1317.98s (#33)	546.11s (#32)	413.46s (#31)
Kimi-K2-thinking	729.12s (#32)	1063.41s (#32)	599.81s (#34)	401.07s (#27)	574.37s (#33)	506.63s (#31)	437.62s (#31)	1223.91s (#31)	1507.26s (#34)	520.56s (#31)	456.51s (#33)
Gpt-5.2-pro	783.82s (#33)	1424.65s (#35)	435.14s (#32)	477.57s (#31)	586.24s (#34)	582.91s (#34)	431.57s (#30)	1706.35s (#34)	1289.05s (#32)	425.52s (#29)	479.20s (#34)
Qwen3-235B-A22B-Thinking-2507	810.89s (#34)	1366.04s (#34)	248.13s (#27)	506.33s (#33)	540.14s (#31)	515.38s (#32)	580.07s (#34)	1677.41s (#33)	1660.09s (#35)	582.45s (#34)	432.82s (#32)
Deepseek-v3.2-speciale	832.67s (#35)	1311.96s (#33)	650.23s (#35)	528.49s (#34)	503.19s (#29)	607.86s (#35)	403.12s (#29)	1755.13s (#35)	1282.71s (#31)	701.66s (#35)	582.35s (#35)