AutoBench Run 2 - April 2025

Second major AutoBench run with o4-mini, GPT-4.1-mini, Gemini 2.5 Pro Preview, Claude 3.7 Sonnet:thinking, etc.

Past

Date

April 25, 2025

Version

2025-04-25

Models

New Models

Run data

Overall Comparison Costs Latency P99 Latency Domain

AutoBench provides a complete dashboard with highly customizable domain-specific average ranks and precise efficiency metrics (average answer cost, average answer duration, P99 Latency). For further detail also check our Hugging Face Leaderboard.

Model	Score	Avg Cost ($ Cents)	Avg Latency (sec)	P99 Latency (sec)	Iterations
claude-3.5-haiku-20241022	3.99 (#22)	0.18 (#15)	11s (#7)	18s (#5)	-
claude-3.7-sonnet	4.39 (#3)	- (#25)	46s (#24)	83s (#20)	-
claude-3.7-sonnet	4.2 (#10)	1.13 (#22)	16s (#12)	33s (#12)	-
deepSeek-R1	4.26 (#7)	0.52 (#17)	85s (#25)	223s (#25)	-
deepSeek-V3	4.09 (#16)	0.09 (#10)	35s (#19)	107s (#23)	-
deepSeek-V3-0324	4.16 (#14)	0.10 (#12)	42s (#22)	141s (#24)	-
gemini-2.0-flash-001	4.16 (#13)	0.04 (#4)	6s (#3)	9s (#1)	-
gemini-2.5-pro-preview-03-25	4.46 (#2)	1.23 (#23)	37s (#21)	64s (#15)	-
gemma-3-27b-it	4.2 (#9)	0.03 (#3)	30s (#16)	79s (#19)	-
gpt-4.1-mini	4.34 (#4)	0.14 (#14)	15s (#11)	29s (#10)	-
gpt-4o-mini	4 (#19)	0.04 (#6)	12s (#9)	22s (#6)	-
grok-2-1212	4.1 (#15)	0.85 (#21)	12s (#8)	23s (#8)	-
grok-3-beta	4.34 (#5)	1.70 (#24)	34s (#18)	70s (#17)	-
llama-3.1-Nemotron-70B-Instruct-HF	4.18 (#11)	0.04 (#7)	25s (#14)	49s (#13)	-
llama-3.3-70B-Instruct	4.02 (#18)	0.04 (#5)	31s (#17)	74s (#18)	-
llama-3_1-Nemotron-Ultra-253B-v1	4.26 (#6)	0.32 (#16)	44s (#23)	94s (#21)	-
llama-4-Maverick-17B-128E-Instruct-FP8	4 (#21)	0.07 (#9)	10s (#5)	23s (#7)	-
llama-4-Scout-17B-16E-Instruct	4 (#20)	0.05 (#8)	8s (#4)	14s (#4)	-
mistral-large-2411	4.05 (#17)	0.53 (#18)	29s (#15)	97s (#22)	-
mistral-small-24b-instruct-2501	3.88 (#24)	0.01 (#1)	14s (#10)	30s (#11)	-
nova-lite-v1	3.89 (#23)	0.02 (#2)	5s (#1)	12s (#3)	-
nova-pro-v1	3.83 (#25)	0.14 (#13)	6s (#2)	10s (#2)	-
o3-mini-2025-01-31	4.26 (#8)	0.61 (#19)	11s (#6)	24s (#9)	-
o4-mini-2025-04-16	4.57 (#1)	0.79 (#20)	19s (#13)	52s (#14)	-
qwen-plus	4.17 (#12)	0.10 (#11)	35s (#20)	67s (#16)	-