AutoBench Run 2 - April 2025

Second major AutoBench run with o4-mini, GPT-4.1-mini, Gemini 2.5 Pro Preview, Claude 3.7 Sonnet:thinking, etc.

Past

Date

April 25, 2025

Version

2025-04-25

Models

New Models

Run data

Overall Comparison Costs Latency P99 Latency Domain

P99 latency breakdown across 10 specialized domains, measured in seconds. This 99th percentile metric reveals worst-case response times, helping you ensure consistent performance for production environments.

Model	Average (All Topics)	Coding	Creative Writing	Current News	General Culture	Grammar	History	Logics	Math	Science	Technology
gemini-2.0-flash-001	8.82s (#1)	13.29s (#2)	6.62s (#2)	7.15s (#2)	7.43s (#2)	9.12s (#3)	6.65s (#2)	10.66s (#3)	12.31s (#1)	7.46s (#2)	7.56s (#1)
nova-pro-v1	9.93s (#2)	15.72s (#3)	11.25s (#5)	6.86s (#1)	10.76s (#3)	7.67s (#2)	8.17s (#3)	9.15s (#2)	14.68s (#2)	6.84s (#1)	8.20s (#2)
nova-lite-v1	12.47s (#3)	12.92s (#1)	5.83s (#1)	32.65s (#13)	6.61s (#1)	6.51s (#1)	5.70s (#1)	9.03s (#1)	19.28s (#3)	7.80s (#3)	18.38s (#9)
llama-4-Scout-17B-16E-Instruct	13.82s (#4)	26.40s (#6)	9.28s (#3)	10.96s (#3)	11.67s (#4)	9.58s (#4)	12.00s (#4)	14.52s (#4)	20.09s (#4)	10.46s (#4)	13.20s (#5)
claude-3.5-haiku-20241022	17.98s (#5)	23.70s (#4)	19.75s (#10)	19.06s (#8)	20.65s (#10)	15.82s (#6)	17.31s (#7)	15.70s (#5)	21.40s (#5)	13.30s (#8)	13.16s (#4)
gpt-4o-mini	21.75s (#6)	31.65s (#7)	18.10s (#9)	16.75s (#7)	19.07s (#7)	20.07s (#10)	22.09s (#10)	17.86s (#6)	37.33s (#6)	17.63s (#9)	16.96s (#8)
llama-4-Maverick-17B-128E-Instruct-FP8	23.11s (#7)	80.03s (#16)	10.11s (#4)	13.50s (#4)	15.19s (#5)	12.64s (#5)	13.25s (#5)	18.17s (#7)	42.55s (#7)	12.47s (#5)	13.21s (#6)
grok-2-1212	23.32s (#8)	25.85s (#5)	12.72s (#6)	13.91s (#5)	17.66s (#6)	17.06s (#8)	17.54s (#8)	28.11s (#8)	72.73s (#11)	13.08s (#6)	14.55s (#7)
o3-mini-2025-01-31	23.67s (#9)	35.06s (#9)	16.40s (#8)	16.14s (#6)	20.01s (#8)	18.56s (#9)	14.95s (#6)	39.78s (#12)	52.40s (#9)	13.09s (#7)	10.28s (#3)
gpt-4.1-mini	29.19s (#10)	39.52s (#11)	15.25s (#7)	20.58s (#9)	32.46s (#12)	25.08s (#12)	28.67s (#11)	36.87s (#11)	52.40s (#10)	19.65s (#10)	21.41s (#11)
mistral-small-24b-instruct-2501	29.62s (#11)	36.17s (#10)	26.90s (#11)	21.63s (#10)	20.29s (#9)	16.93s (#7)	19.97s (#9)	32.45s (#9)	75.08s (#12)	21.31s (#12)	25.44s (#12)
claude-3.7-sonnet	32.86s (#12)	34.50s (#8)	42.15s (#16)	35.20s (#14)	24.45s (#11)	22.00s (#11)	41.91s (#15)	32.66s (#10)	45.31s (#8)	21.00s (#11)	29.41s (#14)
llama-3.1-Nemotron-70B-Instruct-HF	48.74s (#13)	65.57s (#13)	29.35s (#12)	29.17s (#12)	33.06s (#13)	35.17s (#13)	29.16s (#12)	44.70s (#13)	165.37s (#22)	28.79s (#14)	27.03s (#13)
o4-mini-2025-04-16	52.30s (#14)	57.74s (#12)	39.50s (#14)	24.57s (#11)	39.62s (#14)	48.97s (#15)	33.24s (#13)	70.85s (#17)	164.19s (#21)	25.11s (#13)	19.22s (#10)
gemini-2.5-pro-preview-03-25	64.18s (#15)	78.66s (#15)	32.41s (#13)	37.16s (#15)	59.49s (#20)	48.05s (#14)	40.25s (#14)	128.55s (#20)	137.50s (#19)	39.82s (#17)	39.92s (#15)
qwen-plus	66.70s (#16)	77.04s (#14)	72.72s (#23)	55.77s (#20)	55.36s (#18)	69.05s (#21)	48.73s (#17)	68.49s (#15)	121.78s (#18)	41.30s (#18)	56.74s (#17)
grok-3-beta	69.79s (#17)	80.37s (#17)	50.31s (#18)	69.86s (#21)	44.77s (#16)	81.45s (#24)	56.90s (#18)	90.91s (#19)	87.52s (#13)	45.89s (#19)	89.96s (#22)
llama-3.3-70B-Instruct	73.70s (#18)	83.59s (#18)	53.04s (#20)	79.05s (#23)	70.62s (#22)	70.25s (#23)	57.85s (#19)	69.79s (#16)	117.32s (#17)	77.75s (#22)	57.72s (#18)
gemma-3-27b-it	79.12s (#19)	205.53s (#23)	47.10s (#17)	49.13s (#19)	39.73s (#15)	59.89s (#19)	77.24s (#24)	62.05s (#14)	98.21s (#15)	80.09s (#23)	72.23s (#21)
claude-3.7-sonnet	82.60s (#20)	122.38s (#19)	86.02s (#24)	45.39s (#16)	56.93s (#19)	59.81s (#18)	58.50s (#20)	137.09s (#22)	145.15s (#20)	53.35s (#21)	61.38s (#19)
llama-3_1-Nemotron-Ultra-253B-v1	94.45s (#21)	157.34s (#21)	41.92s (#15)	48.86s (#18)	54.35s (#17)	55.23s (#16)	43.90s (#16)	205.29s (#24)	236.74s (#24)	51.17s (#20)	49.75s (#16)
mistral-large-2411	96.77s (#22)	157.72s (#22)	54.36s (#21)	82.25s (#24)	77.68s (#23)	69.55s (#22)	160.87s (#25)	136.52s (#21)	98.81s (#16)	29.76s (#15)	100.24s (#24)
deepSeek-V3	106.53s (#23)	489.39s (#25)	52.40s (#19)	48.32s (#17)	63.66s (#21)	57.52s (#17)	64.71s (#21)	72.51s (#18)	89.72s (#14)	35.64s (#16)	91.46s (#23)
deepSeek-V3-0324	140.54s (#24)	132.59s (#20)	60.61s (#22)	92.63s (#25)	98.62s (#24)	64.37s (#20)	68.91s (#23)	202.20s (#23)	230.86s (#23)	136.28s (#25)	318.33s (#25)
deepSeek-R1	223.47s (#25)	265.87s (#24)	557.13s (#25)	73.00s (#22)	123.61s (#25)	195.61s (#25)	68.46s (#22)	393.58s (#25)	391.41s (#25)	97.07s (#24)	68.91s (#20)