AutoBench Run 3 - August 2025

Latest AutoBench run with enhanced metrics including evaluation iterations and fail rates

Past

Date

August 14, 2025

Version

2025-08-14

Models

New Models

Run data

Overall Comparison Costs Latency P99 Latency Domain

AutoBench provides a complete dashboard with highly customizable domain-specific average ranks and precise efficiency metrics (average answer cost, average answer duration, P99 Latency). For further detail also check our Hugging Face Leaderboard.

Model	Score	Avg Cost ($ Cents)	Avg Latency (sec)	P99 Latency (sec)	Iterations
nova-lite-v1	3.54 (#32)	0.02 (#1)	5s (#1)	10s (#1)	393
phi-4	3.66 (#28)	0.02 (#2)	8s (#3)	19s (#2)	392
gemma-3-27b-it	3.88 (#24)	0.03 (#3)	30s (#12)	135s (#13)	393
llama-4-Scout-17B-16E-Instruct	3.61 (#30)	0.04 (#4)	11s (#5)	40s (#5)	393
Llama-3_3-Nemotron-Super-49B-v1	3.88 (#25)	0.05 (#6)	33s (#13)	151s (#14)	392
llama-4-maverick	3.64 (#29)	0.05 (#5)	11s (#4)	71s (#6)	388
Qwen3-14B	3.98 (#20)	0.08 (#7)	61s (#21)	239s (#23)	392
Qwen3-30B-A3B	3.95 (#22)	0.08 (#8)	73s (#29)	243s (#25)	390
grok-3-mini	4.06 (#17)	0.09 (#9)	26s (#10)	116s (#9)	391
gemini-2.5-flash-lite	4.02 (#18)	0.11 (#10)	19s (#8)	127s (#12)	389
deepSeek-V3-0324	3.95 (#23)	0.12 (#11)	40s (#17)	200s (#19)	392
gpt-oss-120b	4.48 (#3)	0.14 (#12)	27s (#11)	119s (#10)	388
nova-pro-v1	3.49 (#33)	0.18 (#13)	8s (#2)	20s (#3)	389
magistral-small-2506	3.71 (#26)	0.20 (#14)	18s (#7)	90s (#7)	390
Kimi-K2-Instruct	4.18 (#12)	0.24 (#15)	65s (#24)	391s (#33)	325
gpt-5-nano	4.33 (#7)	0.24 (#16)	67s (#27)	232s (#22)	390
llama-3_1-Nemotron-Ultra-253B-v1	4.02 (#19)	0.35 (#17)	62s (#22)	202s (#20)	391
GLM-4.5-Air	3.98 (#21)	0.36 (#18)	68s (#28)	241s (#24)	392
Qwen3-235B-A22B-Thinking-2507	4.39 (#6)	0.42 (#19)	79s (#30)	284s (#32)	331
gemini-2.5-flash	4.32 (#8)	0.45 (#20)	49s (#19)	244s (#26)	387
mistral-large-2411	3.71 (#27)	0.61 (#21)	24s (#9)	97s (#8)	392
GLM-4.5	4.18 (#13)	0.63 (#23)	81s (#31)	246s (#27)	389
gpt-5-mini	4.49 (#2)	0.63 (#22)	66s (#26)	231s (#21)	392
deepSeek-R1-0528	4.18 (#14)	0.64 (#24)	119s (#33)	266s (#29)	385
claude-3.5-haiku	3.59 (#31)	0.83 (#25)	12s (#6)	25s (#4)	393
o4-mini	4.27 (#10)	0.87 (#26)	39s (#16)	186s (#17)	393
gpt-4.1	4.17 (#15)	0.91 (#27)	33s (#14)	181s (#16)	392
gemini-2.5-pro	4.42 (#4)	1.59 (#28)	65s (#25)	199s (#18)	388
claude-sonnet-4	4.17 (#16)	1.71 (#29)	34s (#15)	120s (#11)	393
o3	4.41 (#5)	1.85 (#30)	64s (#23)	277s (#30)	391
grok-4	4.31 (#9)	2.92 (#31)	61s (#20)	263s (#28)	360
gpt-5	4.51 (#1)	4.37 (#32)	90s (#32)	278s (#31)	385
claude-opus-4-1	4.24 (#11)	9.13 (#33)	49s (#18)	155s (#15)	387