AutoBench Run 2 - April 2025

Second major AutoBench run with o4-mini, GPT-4.1-mini, Gemini 2.5 Pro Preview, Claude 3.7 Sonnet:thinking, etc.

Past

Date

April 25, 2025

Version

2025-04-25

Models

New Models

Run data

Overall Comparison Costs Latency P99 Latency Domain

Comparison of AutoBench scores with other popular benchmarks. AutoBench features 92.18% correlation with Artificial Analysis Intelligence Index, 82,22% with LMArena (Chatbot Arena), and 75.45% with MMLU-Plus. Models sorted by AutoBench score.

Model	AutoBench	Chatbot Ar.	AAI Index	MMLU Index
o4-mini-2025-04-16	4.57 (#1)	-	69830 (#1)	0.832 (#4)
gemini-2.5-pro-preview-03-25	4.46 (#2)	1439 (#1)	67840 (#2)	0.858 (#1)
claude-3.7-sonnet	4.39 (#3)	1303 (#10)	57390 (#5)	0.837 (#3)
gpt-4.1-mini	4.34 (#5)	-	52860 (#7)	0.781 (#10)
grok-3-beta	4.34 (#4)	1402 (#2)	50630 (#8)	0.799 (#8)
deepSeek-R1	4.26 (#6)	1358 (#4)	60220 (#4)	0.844 (#2)
llama-3_1-Nemotron-Ultra-253B-v1	4.26 (#8)	-	-	0.69 (#18)
o3-mini-2025-01-31	4.26 (#7)	1305 (#9)	62860 (#3)	0.791 (#9)
claude-3.7-sonnet	4.2 (#10)	1293 (#11)	48150 (#10)	0.803 (#7)
gemma-3-27b-it	4.2 (#9)	1342 (#6)	37620 (#17)	0.669 (#19)
llama-3.1-Nemotron-70B-Instruct-HF	4.18 (#11)	1269 (#15)	37280 (#18)	-
qwen-plus	4.17 (#12)	1310 (#8)	-	-
deepSeek-V3-0324	4.16 (#13)	1372 (#3)	53240 (#6)	0.819 (#5)
gemini-2.0-flash-001	4.16 (#14)	1356 (#5)	48090 (#11)	0.779 (#11)
grok-2-1212	4.1 (#15)	1288 (#12)	39230 (#15)	0.709 (#15)
deepSeek-V3	4.09 (#16)	1318 (#7)	45580 (#12)	0.752 (#12)
mistral-large-2411	4.05 (#17)	1249 (#17)	38270 (#16)	0.697 (#16)
llama-3.3-70B-Instruct	4.02 (#18)	1257 (#16)	41110 (#14)	0.713 (#14)
gpt-4o-mini	4 (#19)	1272 (#13)	35680 (#20)	0.648 (#21)
llama-4-Maverick-17B-128E-Instruct-FP8	4 (#20)	1271 (#14)	50530 (#9)	0.809 (#6)
llama-4-Scout-17B-16E-Instruct	4 (#21)	-	42990 (#13)	0.752 (#13)
claude-3.5-haiku-20241022	3.99 (#22)	1237 (#19)	34740 (#22)	0.634 (#22)
nova-lite-v1	3.89 (#23)	1217 (#21)	32530 (#23)	0.59 (#23)
mistral-small-24b-instruct-2501	3.88 (#24)	1217 (#20)	35280 (#21)	0.652 (#20)
nova-pro-v1	3.83 (#25)	1245 (#18)	37080 (#19)	0.691 (#17)