AutoBench Agentic Run 1 - April 2026

The first AutoBench run to measure agentic performance of top LLMs

Latest

Date

April 19, 2026

Version

2026-04-19

Models

New Models

Run data

Overall Comparison Costs Latency P99 Latency Domain

P99 latency breakdown across 10 specialized domains, measured in seconds. This 99th percentile metric reveals worst-case response times, helping you ensure consistent performance for production environments.

Model	Average (All Topics)	Adaptive Replanning	Api Workflow	Domain Workflow	Error Handling	Failure Recovery	Multi Step Orchestration	Parallel Execution	Parameter Complexity	Single Tool Call	Tool Selection
Mistral large 2512	21.55s (#1)	26.34s (#3)	27.95s (#3)	30.02s (#3)	21.89s (#2)	21.69s (#1)	15.67s (#1)	29.06s (#3)	18.66s (#4)	10.02s (#1)	14.23s (#2)
Gemini 3 Flash Preview	22.53s (#2)	25.08s (#2)	27.29s (#2)	25.37s (#1)	23.12s (#3)	26.88s (#2)	19.43s (#2)	24.81s (#1)	22.61s (#6)	18.78s (#5)	11.92s (#1)
Qwen3.5 35B A3B	30.73s (#3)	34.73s (#5)	51.20s (#8)	34.91s (#5)	37.86s (#5)	30.83s (#3)	29.59s (#4)	43.74s (#4)	13.65s (#2)	14.12s (#3)	16.67s (#5)
Qwen3.5 122B A10B	36.45s (#4)	40.97s (#7)	34.92s (#5)	36.33s (#6)	30.20s (#4)	46.45s (#4)	32.10s (#5)	78.24s (#9)	17.63s (#3)	25.20s (#8)	22.47s (#7)
Gemma 4 26B A4B IT	41.00s (#5)	28.19s (#4)	38.43s (#6)	33.85s (#4)	46.89s (#6)	47.65s (#5)	47.89s (#8)	106.97s (#14)	22.32s (#5)	22.47s (#6)	15.30s (#4)
Mistral Small 4	41.25s (#6)	22.59s (#1)	18.28s (#1)	26.93s (#2)	17.24s (#1)	243.06s (#27)	20.71s (#3)	26.23s (#2)	8.45s (#1)	11.09s (#2)	17.96s (#6)
Claude Opus 4.7	47.01s (#7)	37.49s (#6)	63.41s (#11)	58.66s (#9)	64.33s (#11)	47.68s (#6)	33.14s (#6)	86.67s (#10)	33.18s (#7)	17.99s (#4)	27.53s (#8)
Mimo V2 Pro	54.67s (#8)	45.97s (#9)	69.68s (#12)	43.21s (#7)	53.61s (#7)	57.12s (#7)	64.15s (#11)	72.84s (#7)	49.35s (#12)	61.98s (#17)	28.82s (#9)
MiniMax M2.7	57.54s (#9)	70.46s (#13)	63.29s (#10)	70.59s (#12)	73.86s (#13)	64.28s (#8)	60.17s (#10)	62.53s (#5)	45.93s (#10)	32.23s (#9)	32.07s (#10)
Gemini 3.1 Pro Preview	58.21s (#10)	44.94s (#8)	53.45s (#9)	43.39s (#8)	55.13s (#9)	86.87s (#14)	51.50s (#9)	113.27s (#15)	46.15s (#11)	43.25s (#11)	44.14s (#12)
Gpt oss 120b	63.36s (#11)	51.60s (#10)	33.31s (#4)	73.13s (#13)	66.80s (#12)	76.08s (#11)	43.41s (#7)	91.52s (#12)	56.49s (#14)	126.51s (#24)	14.77s (#3)
Llama 4 Maverick	76.10s (#12)	64.54s (#12)	73.80s (#13)	67.02s (#11)	63.99s (#10)	82.16s (#12)	118.40s (#18)	76.50s (#8)	72.38s (#17)	68.62s (#20)	73.61s (#21)
Grok 4.20	78.57s (#13)	60.92s (#11)	180.76s (#20)	61.16s (#10)	111.80s (#16)	70.48s (#9)	86.79s (#12)	72.18s (#6)	60.30s (#15)	40.20s (#10)	41.12s (#11)
Grok 4.1 fast	95.82s (#14)	98.08s (#17)	86.55s (#14)	110.61s (#18)	54.03s (#8)	88.82s (#15)	236.05s (#26)	86.72s (#11)	73.69s (#18)	56.25s (#15)	67.38s (#19)
Claude Opus 4.6	97.92s (#15)	84.54s (#15)	197.75s (#26)	100.16s (#17)	80.42s (#14)	86.53s (#13)	111.44s (#17)	150.53s (#19)	70.63s (#16)	44.96s (#14)	52.25s (#14)
Qwen3.6 Plus	106.88s (#16)	99.61s (#18)	113.88s (#18)	180.36s (#22)	163.17s (#19)	75.95s (#10)	90.04s (#14)	114.39s (#16)	43.05s (#8)	90.51s (#23)	97.86s (#24)
Gemini 3.1 Flash Lite Preview	114.39s (#17)	158.95s (#21)	43.21s (#7)	83.20s (#15)	191.29s (#22)	237.75s (#26)	119.98s (#19)	140.97s (#17)	86.19s (#19)	24.62s (#7)	57.78s (#17)
Deepseek v3.2	129.25s (#18)	188.98s (#25)	112.56s (#17)	79.21s (#14)	88.07s (#15)	259.09s (#29)	250.29s (#28)	152.20s (#20)	49.59s (#13)	56.83s (#16)	55.67s (#16)
GLM 4.7	134.63s (#19)	129.62s (#19)	110.14s (#15)	95.46s (#16)	303.95s (#29)	131.78s (#17)	89.62s (#13)	212.00s (#24)	160.59s (#27)	63.75s (#18)	49.37s (#13)
Nova 2 lite v1	138.83s (#20)	174.36s (#23)	185.77s (#22)	131.29s (#20)	142.23s (#17)	201.72s (#22)	135.57s (#20)	185.40s (#22)	118.55s (#23)	43.37s (#12)	70.04s (#20)
Claude Sonnet 4.6	149.12s (#21)	154.34s (#20)	191.42s (#24)	279.76s (#30)	153.42s (#18)	208.83s (#24)	106.87s (#16)	141.66s (#18)	147.49s (#26)	43.59s (#13)	63.82s (#18)
Claude haiku 4.5	151.05s (#22)	80.59s (#14)	110.27s (#16)	127.08s (#19)	276.19s (#27)	113.23s (#16)	241.89s (#27)	95.15s (#13)	97.59s (#20)	78.06s (#22)	290.47s (#30)
Gpt oss 20b	165.52s (#23)	185.96s (#24)	182.14s (#21)	169.19s (#21)	208.18s (#24)	174.71s (#19)	141.33s (#21)	205.74s (#23)	137.31s (#24)	145.45s (#25)	105.24s (#25)
Gemma 4 31B IT	174.19s (#24)	94.18s (#16)	192.00s (#25)	250.36s (#26)	188.65s (#21)	168.30s (#18)	141.91s (#22)	283.42s (#29)	43.56s (#9)	65.94s (#19)	313.53s (#31)
Kimi K2.5	177.55s (#25)	334.16s (#30)	122.39s (#19)	189.85s (#23)	167.28s (#20)	191.31s (#20)	97.54s (#15)	274.55s (#26)	139.53s (#25)	177.26s (#26)	81.63s (#22)
GLM 5.1	183.27s (#26)	167.48s (#22)	251.43s (#30)	275.43s (#29)	192.38s (#23)	233.09s (#25)	168.08s (#23)	281.39s (#27)	104.81s (#21)	72.48s (#21)	86.12s (#23)
MiniMax M2.5	240.81s (#27)	240.50s (#26)	217.96s (#28)	268.45s (#27)	395.57s (#32)	245.20s (#28)	268.93s (#29)	238.99s (#25)	112.89s (#22)	200.63s (#27)	219.00s (#28)
GPT-5.4 Mini (xhigh)	240.88s (#28)	333.09s (#29)	217.50s (#27)	303.03s (#32)	239.12s (#25)	302.55s (#32)	198.36s (#24)	179.33s (#21)	267.11s (#30)	313.20s (#30)	55.52s (#15)
Nemotron 3 Super 120B A12B	245.21s (#29)	346.35s (#31)	232.45s (#29)	206.61s (#25)	305.13s (#30)	295.02s (#31)	227.88s (#25)	315.69s (#30)	169.17s (#28)	212.15s (#28)	141.62s (#26)
GPT-5.4 Nano (xhigh)	262.21s (#30)	254.31s (#27)	188.87s (#23)	196.32s (#24)	291.05s (#28)	196.74s (#21)	301.91s (#30)	327.64s (#32)	320.11s (#31)	320.85s (#31)	224.27s (#29)
Nemotron 3 Nano 30B A3B	284.56s (#31)	312.20s (#28)	357.31s (#32)	290.42s (#31)	252.05s (#26)	278.36s (#30)	387.20s (#32)	282.72s (#28)	200.52s (#29)	267.32s (#29)	217.48s (#27)
GPT-5.4 (xhigh)	325.77s (#32)	356.84s (#32)	304.61s (#31)	268.49s (#28)	393.60s (#31)	205.17s (#23)	323.54s (#31)	321.91s (#31)	396.14s (#32)	334.06s (#32)	353.36s (#32)