AutoBench Run 3 - August 2025

Latest AutoBench run with enhanced metrics including evaluation iterations and fail rates

Latest

Date

August 14, 2025

Version

2025-08-14

Models

New Models

Run data

Overall Comparison Costs Latency P99 Latency Domain

P99 latency breakdown across 10 specialized domains, measured in seconds. This 99th percentile metric reveals worst-case response times, helping you ensure consistent performance for production environments.

Model	Average (All Topics)	Coding	Creative Writing	Current News	General Culture	Grammar	History	Logics	Math	Science	Technology
nova-pro-v1	20.16s (#3)	55.83s (#5)	13.38s (#5)	14.29s (#3)	9.77s (#1)	24.34s (#8)	15.61s (#3)	14.29s (#3)	23.76s (#2)	15.25s (#3)	15.04s (#2)
nova-lite-v1	10.28s (#1)	17.36s (#1)	9.04s (#2)	11.67s (#1)	10.19s (#2)	8.04s (#1)	9.78s (#1)	8.27s (#1)	7.75s (#1)	9.47s (#1)	11.20s (#1)
gemini-2.5-flash-lite	127.44s (#12)	190.82s (#8)	6.29s (#1)	19.33s (#4)	12.16s (#3)	13.28s (#2)	47.89s (#8)	602.47s (#16)	222.26s (#15)	49.20s (#7)	110.74s (#14)
phi-4	19.19s (#2)	28.12s (#2)	10.37s (#3)	12.39s (#2)	13.48s (#4)	13.46s (#3)	12.35s (#2)	14.12s (#2)	39.95s (#4)	13.62s (#2)	34.03s (#7)
llama-4-maverick	71.14s (#6)	258.29s (#12)	12.31s (#4)	28.32s (#7)	14.16s (#5)	15.95s (#6)	23.47s (#5)	237.75s (#10)	50.10s (#6)	44.60s (#6)	26.43s (#5)
llama-4-Scout-17B-16E-Instruct	39.62s (#5)	119.44s (#6)	17.91s (#7)	21.72s (#5)	15.14s (#6)	15.59s (#5)	21.44s (#4)	21.64s (#5)	35.70s (#3)	109.50s (#9)	18.14s (#4)
magistral-small-2506	89.52s (#7)	50.67s (#4)	23.69s (#9)	23.00s (#6)	17.83s (#7)	14.26s (#4)	27.63s (#6)	461.29s (#14)	227.31s (#16)	22.41s (#5)	27.14s (#6)
claude-3.5-haiku	25.35s (#4)	41.31s (#3)	14.95s (#6)	33.58s (#8)	18.55s (#8)	17.73s (#7)	36.65s (#7)	15.82s (#4)	40.06s (#5)	17.90s (#4)	16.93s (#3)
grok-3-mini	116.10s (#9)	324.73s (#19)	28.87s (#13)	38.36s (#9)	27.44s (#9)	58.73s (#13)	56.70s (#10)	303.21s (#12)	79.71s (#9)	164.64s (#18)	78.63s (#11)
gpt-4.1	180.74s (#16)	373.04s (#25)	20.97s (#8)	103.65s (#23)	34.64s (#10)	45.12s (#10)	68.03s (#13)	580.15s (#15)	268.42s (#19)	151.75s (#15)	161.64s (#19)
gemini-2.5-flash	244.11s (#26)	712.32s (#33)	38.33s (#14)	103.80s (#24)	43.56s (#11)	74.99s (#20)	117.56s (#22)	944.40s (#29)	122.86s (#12)	135.36s (#10)	147.92s (#17)
o4-mini	185.54s (#17)	317.20s (#17)	49.14s (#18)	78.87s (#14)	45.30s (#12)	67.70s (#16)	52.90s (#9)	768.08s (#24)	246.01s (#18)	143.24s (#11)	86.98s (#12)
gemma-3-27b-it	134.51s (#13)	375.59s (#26)	26.32s (#10)	60.81s (#10)	46.14s (#13)	66.88s (#15)	99.04s (#16)	180.09s (#9)	228.18s (#17)	193.23s (#26)	68.86s (#8)
GLM-4.5-Air	240.50s (#24)	294.81s (#14)	49.60s (#19)	75.29s (#11)	47.32s (#14)	188.03s (#31)	122.75s (#25)	934.91s (#28)	326.18s (#22)	192.68s (#25)	173.42s (#24)
mistral-large-2411	96.89s (#8)	320.72s (#18)	28.58s (#12)	76.11s (#12)	50.58s (#15)	34.03s (#9)	104.24s (#18)	69.13s (#7)	52.99s (#7)	161.37s (#17)	71.10s (#9)
claude-sonnet-4	119.60s (#11)	372.49s (#24)	48.28s (#17)	92.44s (#19)	50.91s (#16)	54.40s (#12)	124.80s (#26)	53.75s (#6)	57.44s (#8)	181.60s (#23)	159.87s (#18)
llama-3_1-Nemotron-Ultra-253B-v1	201.97s (#20)	299.24s (#15)	64.72s (#27)	80.61s (#15)	53.24s (#17)	111.74s (#25)	114.96s (#20)	677.22s (#20)	364.49s (#28)	179.97s (#22)	73.47s (#10)
Llama-3_3-Nemotron-Super-49B-v1	151.44s (#14)	215.99s (#9)	28.14s (#11)	78.59s (#13)	55.05s (#18)	69.65s (#17)	64.65s (#11)	672.34s (#19)	82.64s (#10)	150.75s (#14)	96.61s (#13)
gemini-2.5-pro	199.34s (#18)	240.77s (#11)	52.38s (#21)	97.35s (#21)	57.95s (#19)	71.32s (#18)	111.13s (#19)	714.43s (#22)	300.57s (#20)	170.19s (#20)	177.33s (#25)
gpt-oss-120b	119.15s (#10)	219.91s (#10)	41.00s (#15)	88.25s (#17)	59.79s (#20)	50.04s (#11)	66.35s (#12)	154.42s (#8)	213.96s (#14)	154.40s (#16)	143.39s (#16)
GLM-4.5	246.05s (#27)	336.58s (#22)	46.15s (#16)	115.42s (#28)	61.51s (#21)	183.87s (#30)	94.37s (#14)	955.21s (#31)	354.67s (#27)	147.68s (#13)	164.99s (#20)
gpt-5-mini	231.38s (#21)	420.19s (#29)	55.51s (#22)	107.65s (#25)	65.40s (#22)	94.14s (#22)	96.48s (#15)	710.37s (#21)	304.14s (#21)	221.41s (#30)	238.51s (#30)
claude-opus-4-1	155.15s (#15)	411.82s (#28)	66.28s (#28)	99.78s (#22)	67.81s (#23)	73.13s (#19)	140.07s (#31)	240.51s (#11)	85.49s (#11)	194.42s (#27)	172.18s (#23)
deepSeek-V3-0324	199.71s (#19)	186.89s (#7)	51.30s (#20)	88.11s (#16)	69.85s (#24)	65.72s (#14)	121.31s (#24)	755.24s (#23)	202.09s (#13)	100.63s (#8)	355.96s (#33)
deepSeek-R1-0528	265.55s (#29)	516.75s (#31)	74.29s (#30)	114.09s (#27)	72.93s (#25)	112.01s (#26)	127.05s (#27)	839.46s (#25)	432.45s (#30)	182.40s (#24)	184.12s (#27)
grok-4	262.52s (#28)	330.67s (#21)	74.00s (#29)	112.44s (#26)	75.37s (#26)	148.48s (#28)	118.40s (#23)	908.29s (#26)	484.36s (#31)	205.04s (#29)	168.17s (#22)
Qwen3-30B-A3B	243.07s (#25)	302.32s (#16)	64.12s (#26)	92.98s (#20)	77.85s (#27)	121.60s (#27)	100.32s (#17)	973.63s (#33)	352.38s (#25)	165.05s (#19)	180.50s (#26)
o3	276.73s (#30)	370.63s (#23)	215.20s (#33)	126.72s (#30)	84.40s (#28)	96.71s (#23)	130.47s (#28)	970.11s (#32)	427.76s (#29)	179.88s (#21)	165.46s (#21)
gpt-5-nano	231.92s (#22)	452.68s (#30)	63.87s (#25)	123.40s (#29)	95.27s (#29)	98.38s (#24)	131.31s (#30)	649.72s (#17)	349.24s (#24)	145.60s (#12)	209.74s (#28)
gpt-5	277.67s (#31)	379.02s (#27)	104.18s (#31)	151.02s (#31)	109.28s (#30)	163.05s (#29)	141.32s (#32)	655.51s (#18)	536.53s (#33)	304.23s (#32)	232.58s (#29)
Qwen3-14B	239.24s (#23)	291.19s (#13)	58.06s (#23)	88.56s (#18)	117.73s (#31)	88.07s (#21)	115.43s (#21)	952.76s (#30)	353.03s (#26)	203.27s (#28)	124.24s (#15)
Qwen3-235B-A22B-Thinking-2507	283.84s (#32)	666.74s (#32)	62.61s (#24)	184.42s (#32)	141.18s (#32)	188.18s (#32)	130.83s (#29)	431.52s (#13)	488.49s (#32)	231.81s (#31)	312.64s (#32)
Kimi-K2-Instruct	390.47s (#33)	328.17s (#20)	127.06s (#32)	498.21s (#33)	155.91s (#33)	411.22s (#33)	374.84s (#33)	919.13s (#27)	342.63s (#23)	464.49s (#33)	283.02s (#31)