AutoBench Run 3 - August 2025

Latest AutoBench run with enhanced metrics including evaluation iterations and fail rates

Latest

Date

August 14, 2025

Version

2025-08-14

Models

New Models

Run data

Overall Comparison Costs Latency P99 Latency Domain

Average latency breakdown across 10 specialized domains, measured in seconds. These metrics help you identify models that consistently deliver quick responses for your specific use case while maintaining quality standards.

Model	Average (All Topics)	Coding	Creative Writing	Current News	General Culture	Grammar	History	Logics	Math	Science	Technology
gpt-5	90.00s (#32)	120.97s (#30)	50.04s (#33)	78.84s (#33)	55.10s (#33)	56.65s (#33)	73.40s (#32)	156.60s (#24)	151.20s (#30)	83.20s (#33)	76.55s (#33)
Kimi-K2-Instruct	65.02s (#24)	69.47s (#18)	29.26s (#28)	65.40s (#32)	45.96s (#31)	46.64s (#29)	75.24s (#33)	114.36s (#18)	58.67s (#20)	72.55s (#32)	57.17s (#30)
Qwen3-235B-A22B-Thinking-2507	78.79s (#30)	180.14s (#32)	33.74s (#30)	65.22s (#31)	45.30s (#30)	54.66s (#32)	53.71s (#28)	122.66s (#20)	138.29s (#28)	60.43s (#30)	72.91s (#32)
deepSeek-R1-0528	119.17s (#33)	238.20s (#33)	38.15s (#31)	62.97s (#30)	48.22s (#32)	51.71s (#31)	61.99s (#31)	302.30s (#33)	271.52s (#33)	62.54s (#31)	66.13s (#31)
gpt-5-nano	66.50s (#27)	98.42s (#26)	38.22s (#32)	52.23s (#29)	38.22s (#29)	48.28s (#30)	54.48s (#30)	136.66s (#22)	86.48s (#22)	48.33s (#26)	53.91s (#28)
grok-4	60.96s (#20)	92.70s (#23)	28.46s (#25)	49.77s (#28)	33.72s (#26)	41.45s (#27)	48.44s (#27)	138.43s (#23)	124.39s (#25)	48.72s (#27)	50.10s (#25)
gemini-2.5-pro	65.03s (#25)	83.39s (#21)	29.26s (#27)	49.16s (#27)	36.70s (#28)	36.99s (#24)	47.21s (#26)	166.92s (#26)	93.46s (#23)	49.47s (#28)	52.59s (#27)
gpt-5-mini	65.90s (#26)	102.52s (#27)	25.72s (#22)	48.42s (#26)	31.52s (#23)	35.42s (#23)	44.41s (#25)	159.72s (#25)	84.27s (#21)	52.17s (#29)	56.72s (#29)
o3	63.90s (#23)	70.42s (#19)	25.94s (#23)	46.46s (#25)	29.61s (#22)	26.53s (#19)	42.66s (#24)	194.81s (#28)	112.94s (#24)	41.25s (#23)	46.78s (#23)
claude-opus-4-1	48.62s (#18)	97.99s (#25)	31.75s (#29)	44.91s (#24)	36.35s (#27)	30.17s (#20)	53.86s (#29)	38.46s (#11)	32.15s (#9)	48.13s (#25)	52.28s (#26)
GLM-4.5	80.74s (#31)	147.04s (#31)	20.92s (#21)	43.49s (#23)	31.81s (#24)	38.41s (#25)	42.51s (#23)	224.90s (#32)	165.52s (#32)	43.16s (#24)	49.76s (#24)
claude-sonnet-4	33.67s (#15)	75.45s (#20)	18.12s (#19)	34.98s (#22)	23.57s (#20)	18.97s (#12)	35.19s (#20)	19.96s (#7)	24.19s (#7)	36.59s (#18)	34.59s (#18)
Qwen3-30B-A3B	72.64s (#29)	119.99s (#29)	27.91s (#24)	34.78s (#21)	25.85s (#21)	38.81s (#26)	37.06s (#22)	204.23s (#30)	157.67s (#31)	38.20s (#20)	41.63s (#20)
gemini-2.5-flash	48.71s (#19)	95.88s (#24)	16.73s (#17)	32.10s (#20)	18.35s (#14)	24.89s (#18)	29.12s (#16)	133.86s (#21)	52.62s (#19)	31.52s (#15)	36.19s (#19)
deepSeek-V3-0324	40.30s (#17)	55.10s (#13)	17.82s (#18)	32.07s (#19)	23.04s (#19)	24.28s (#17)	31.00s (#18)	88.90s (#17)	51.26s (#18)	29.38s (#14)	43.42s (#21)
Qwen3-14B	61.12s (#21)	67.73s (#17)	19.92s (#20)	31.32s (#18)	32.18s (#25)	32.24s (#22)	31.20s (#19)	197.42s (#29)	132.55s (#26)	40.17s (#22)	31.52s (#16)
GLM-4.5-Air	68.34s (#28)	105.91s (#28)	13.72s (#13)	31.22s (#17)	19.54s (#15)	45.69s (#28)	30.92s (#17)	206.30s (#31)	140.95s (#29)	38.18s (#19)	44.16s (#22)
llama-3_1-Nemotron-Ultra-253B-v1	61.54s (#22)	88.51s (#22)	28.89s (#26)	29.22s (#16)	22.75s (#17)	32.21s (#21)	36.15s (#21)	174.68s (#27)	134.89s (#27)	38.29s (#21)	32.33s (#17)
gpt-oss-120b	27.01s (#11)	49.98s (#11)	11.36s (#9)	28.99s (#15)	18.29s (#12)	14.40s (#10)	22.62s (#10)	25.45s (#10)	36.60s (#12)	27.85s (#12)	28.81s (#15)
Llama-3_3-Nemotron-Super-49B-v1	32.64s (#13)	46.36s (#10)	15.18s (#15)	28.78s (#14)	22.91s (#18)	21.54s (#15)	27.26s (#15)	67.43s (#13)	33.04s (#11)	31.87s (#16)	25.02s (#13)
gemma-3-27b-it	29.72s (#12)	62.27s (#16)	12.47s (#11)	27.51s (#13)	18.32s (#13)	19.29s (#13)	25.02s (#13)	24.37s (#9)	40.47s (#14)	35.35s (#17)	23.28s (#10)
o4-mini	39.05s (#16)	56.98s (#14)	16.40s (#16)	26.73s (#12)	19.81s (#16)	21.71s (#16)	23.26s (#12)	116.34s (#19)	41.53s (#16)	28.65s (#13)	26.42s (#14)
gpt-4.1	32.86s (#14)	58.02s (#15)	11.49s (#10)	24.72s (#11)	14.47s (#10)	15.96s (#11)	23.16s (#11)	80.41s (#15)	46.81s (#17)	21.99s (#9)	23.40s (#11)
mistral-large-2411	24.36s (#9)	51.77s (#12)	14.28s (#14)	23.60s (#10)	17.35s (#11)	13.37s (#9)	25.84s (#14)	18.14s (#6)	24.62s (#8)	25.15s (#10)	21.90s (#9)
grok-3-mini	26.12s (#10)	45.77s (#9)	12.96s (#12)	17.72s (#9)	13.61s (#9)	20.44s (#14)	20.45s (#9)	40.52s (#12)	32.54s (#10)	25.81s (#11)	23.90s (#12)
claude-3.5-haiku	11.52s (#6)	17.21s (#5)	9.50s (#8)	13.16s (#8)	11.23s (#8)	9.27s (#8)	13.71s (#8)	7.91s (#4)	9.50s (#3)	10.84s (#7)	11.28s (#8)
llama-4-Scout-17B-16E-Instruct	10.87s (#5)	20.22s (#6)	6.43s (#5)	9.19s (#7)	7.53s (#7)	7.99s (#7)	9.81s (#7)	11.12s (#5)	13.07s (#6)	12.25s (#8)	8.23s (#6)
llama-4-maverick	10.65s (#4)	21.57s (#7)	4.76s (#2)	7.82s (#6)	5.84s (#5)	6.17s (#5)	8.32s (#5)	21.06s (#8)	11.33s (#4)	8.72s (#6)	6.83s (#3)
nova-pro-v1	7.53s (#2)	12.48s (#4)	7.58s (#7)	7.52s (#5)	5.67s (#3)	6.53s (#6)	7.34s (#4)	5.86s (#2)	7.27s (#2)	6.68s (#3)	6.78s (#2)
magistral-small-2506	17.54s (#7)	11.46s (#3)	7.20s (#6)	7.52s (#4)	5.75s (#4)	6.11s (#4)	8.70s (#6)	79.27s (#14)	37.96s (#13)	7.09s (#5)	7.28s (#4)
phi-4	7.74s (#3)	10.84s (#2)	5.95s (#4)	6.78s (#3)	6.31s (#6)	6.00s (#3)	7.15s (#3)	7.76s (#3)	12.17s (#5)	7.04s (#4)	7.41s (#5)
nova-lite-v1	5.29s (#1)	7.10s (#1)	4.79s (#3)	5.85s (#2)	4.71s (#2)	4.34s (#2)	5.51s (#1)	4.88s (#1)	4.86s (#1)	4.91s (#1)	5.33s (#1)
gemini-2.5-flash-lite	19.16s (#8)	26.18s (#8)	2.86s (#1)	5.62s (#1)	3.62s (#1)	3.98s (#1)	6.84s (#2)	86.53s (#16)	41.51s (#15)	6.41s (#2)	9.55s (#7)