AutoBench Run 4 - November 2025

Latest AutoBench run with models Gemini 3 Pro, Gpt 5.1, Grok 4.1 and more

Past

Date

November 28, 2025

Version

2025-11-28

Models

New Models

Run data

Overall Comparison Costs Latency P99 Latency Domain

AutoBench provides a complete dashboard with highly customizable domain-specific average ranks and precise efficiency metrics (average answer cost, average answer duration, P99 Latency). For further detail also check our Hugging Face Leaderboard.

Model	Score	Avg Cost ($ Cents)	Avg Latency (sec)	P99 Latency (sec)	Iterations
gpt-5.1	4.49 (#1)	7.53 (#30)	130s (#30)	386s (#29)	328
gpt-5	4.45 (#2)	7.71 (#31)	152s (#32)	434s (#32)	328
gemini-3-pro-preview	4.39 (#3)	4.62 (#28)	45s (#20)	119s (#13)	324
gemini-2.5-pro	4.37 (#5)	4.84 (#29)	55s (#21)	170s (#18)	329
gpt-oss-120b	4.37 (#4)	0.06 (#8)	17s (#6)	55s (#7)	329
Kimi-K2-thinking	4.34 (#7)	0.98 (#24)	86s (#25)	429s (#31)	328
grok-4.1-fast-thinking	4.34 (#6)	0.19 (#17)	64s (#23)	223s (#23)	329
gpt-5-nano	4.32 (#8)	0.26 (#18)	69s (#24)	207s (#20)	329
claude-sonnet-4.5	4.31 (#9)	4.06 (#27)	58s (#22)	186s (#19)	326
gemini-2.5-flash	4.3 (#10)	1.42 (#25)	40s (#15)	154s (#16)	329
Qwen3-235B-A22B-Thinking-2507	4.28 (#11)	0.18 (#16)	136s (#31)	421s (#30)	319
claude-haiku-4.5	4.27 (#12)	1.88 (#26)	42s (#16)	166s (#17)	330
claude-opus-4-1	4.27 (#13)	15.44 (#32)	114s (#29)	362s (#27)	324
GLM-4.6	4.25 (#14)	0.83 (#21)	100s (#27)	362s (#26)	327
Qwen3-235B-A22B-2507	4.24 (#15)	0.29 (#19)	43s (#17)	223s (#22)	330
deepSeek-R1-0528	4.23 (#16)	0.58 (#20)	107s (#28)	348s (#25)	325
gemini-2.5-flash-lite	4.22 (#17)	0.14 (#13)	14s (#4)	69s (#10)	330
Kimi-K2-0905	4.21 (#19)	0.17 (#15)	45s (#19)	220s (#21)	328
Qwen3-30B-A3B-Instruct-2507	4.21 (#18)	0.05 (#7)	38s (#14)	131s (#15)	331
grok-4.1-fast	4.17 (#20)	0.07 (#9)	23s (#11)	66s (#9)	327
deepSeek-v3.2-exp	4.16 (#21)	0.12 (#11)	100s (#26)	384s (#28)	328
llama-3.3-nemotron-super-49b-v1.5	4.1 (#22)	0.13 (#12)	44s (#18)	128s (#14)	331
grok-3-mini	4.08 (#23)	0.10 (#10)	20s (#10)	53s (#6)	331
magistral-medium-2506	3.95 (#24)	0.89 (#22)	17s (#7)	57s (#8)	331
mistral-small-3.2-24b-instruct	3.81 (#25)	0.02 (#2)	15s (#5)	49s (#5)	332
gemma-3-27b-it	3.7 (#26)	0.03 (#3)	36s (#13)	252s (#24)	326
nova-premier-v1	3.7 (#27)	0.91 (#23)	13s (#3)	33s (#2)	330
llama-4-maverick	3.61 (#28)	0.05 (#6)	13s (#2)	48s (#4)	330
nemotron-nano-9b-v2	3.6 (#29)	0.04 (#5)	25s (#12)	105s (#12)	331
llama-3.3-70B-Instruct	3.55 (#30)	0.04 (#4)	18s (#9)	79s (#11)	331
phi-4	3.46 (#31)	0.01 (#1)	18s (#8)	47s (#3)	331
nova-pro-v1	3.35 (#32)	0.17 (#14)	7s (#1)	21s (#1)	330