AutoBench Run 4 - November 2025

Latest AutoBench run with models Gemini 3 Pro, Gpt 5.1, Grok 4.1 and more

Past

Date

November 28, 2025

Version

2025-11-28

Models

New Models

Run data

Overall Comparison Costs Latency P99 Latency Domain

P99 latency breakdown across 10 specialized domains, measured in seconds. This 99th percentile metric reveals worst-case response times, helping you ensure consistent performance for production environments.

Model	Average (All Topics)	Coding	Creative Writing	Current News	General Culture	Grammar	History	Logics	Math	Science	Technology
nova-pro-v1	21.14s (#1)	25.28s (#1)	26.02s (#2)	14.35s (#1)	11.88s (#1)	11.12s (#1)	13.90s (#1)	7.32s (#1)	8.00s (#1)	13.48s (#1)	80.04s (#15)
llama-4-maverick	48.30s (#4)	58.72s (#2)	46.82s (#7)	37.60s (#8)	40.22s (#9)	70.61s (#11)	29.89s (#8)	43.15s (#4)	65.04s (#8)	60.57s (#5)	30.35s (#5)
mistral-small-3.2-24b-instruct	48.67s (#5)	67.15s (#3)	43.87s (#6)	34.54s (#6)	23.85s (#5)	54.65s (#9)	30.80s (#9)	78.34s (#5)	81.63s (#9)	31.98s (#3)	39.89s (#7)
phi-4	47.27s (#3)	69.84s (#4)	50.69s (#8)	31.22s (#5)	23.58s (#4)	56.30s (#10)	25.30s (#6)	35.51s (#3)	45.54s (#4)	75.56s (#12)	59.20s (#11)
gpt-oss-120b	55.04s (#7)	75.69s (#5)	54.47s (#9)	62.09s (#14)	45.32s (#11)	49.10s (#7)	54.44s (#14)	82.29s (#6)	28.16s (#3)	67.11s (#7)	31.70s (#6)
nova-premier-v1	33.34s (#2)	77.37s (#6)	28.83s (#3)	22.43s (#3)	18.92s (#2)	23.31s (#2)	24.10s (#4)	21.66s (#2)	18.83s (#2)	72.99s (#11)	24.94s (#1)
grok-3-mini	52.79s (#6)	79.39s (#7)	62.99s (#10)	37.28s (#7)	22.81s (#3)	45.51s (#5)	26.27s (#7)	82.48s (#7)	61.29s (#7)	52.98s (#4)	56.86s (#9)
grok-4.1-fast	65.97s (#9)	86.21s (#8)	42.75s (#5)	48.11s (#9)	39.46s (#8)	47.03s (#6)	36.56s (#11)	181.20s (#13)	47.92s (#5)	71.83s (#9)	58.63s (#10)
gemini-2.5-flash-lite	68.94s (#10)	107.49s (#9)	63.52s (#11)	29.32s (#4)	111.23s (#20)	32.54s (#4)	24.96s (#5)	147.52s (#10)	48.25s (#6)	81.00s (#13)	43.56s (#8)
magistral-medium-2506	57.38s (#8)	109.37s (#10)	19.31s (#1)	91.22s (#19)	25.94s (#6)	30.73s (#3)	16.73s (#2)	93.75s (#9)	136.71s (#15)	23.73s (#2)	26.33s (#3)
llama-3.3-70B-Instruct	79.45s (#11)	115.83s (#11)	111.30s (#16)	60.42s (#13)	53.14s (#15)	52.31s (#8)	113.02s (#21)	91.53s (#8)	106.51s (#11)	62.98s (#6)	27.48s (#4)
gemini-3-pro-preview	118.64s (#13)	167.60s (#12)	99.80s (#15)	53.90s (#11)	48.02s (#12)	117.17s (#15)	79.41s (#18)	211.76s (#15)	136.35s (#14)	200.95s (#19)	71.44s (#14)
gemini-2.5-flash	154.24s (#16)	181.89s (#13)	93.20s (#13)	81.59s (#16)	48.40s (#13)	88.19s (#12)	43.76s (#12)	164.87s (#11)	190.15s (#19)	465.01s (#28)	185.30s (#20)
claude-haiku-4.5	165.87s (#17)	231.41s (#14)	182.21s (#23)	124.52s (#23)	61.55s (#17)	195.64s (#23)	81.99s (#19)	240.03s (#20)	106.39s (#10)	93.34s (#15)	341.59s (#30)
Qwen3-30B-A3B-Instruct-2507	131.29s (#15)	272.03s (#15)	125.77s (#18)	51.69s (#10)	37.71s (#7)	139.73s (#18)	78.44s (#17)	239.30s (#19)	213.12s (#20)	72.08s (#10)	83.05s (#16)
Kimi-K2-0905	220.47s (#21)	273.43s (#16)	159.63s (#20)	119.43s (#22)	145.50s (#24)	108.31s (#13)	132.90s (#25)	449.99s (#23)	134.48s (#13)	428.59s (#25)	252.49s (#22)
gemma-3-27b-it	252.24s (#24)	297.55s (#17)	379.46s (#31)	87.32s (#17)	422.39s (#32)	129.06s (#16)	278.20s (#32)	411.45s (#22)	158.62s (#16)	71.57s (#8)	286.80s (#28)
llama-3.3-nemotron-super-49b-v1.5	128.06s (#14)	304.31s (#18)	65.61s (#12)	60.32s (#12)	55.75s (#16)	143.34s (#19)	32.30s (#10)	212.31s (#16)	260.34s (#22)	86.74s (#14)	59.65s (#12)
grok-4.1-fast-thinking	223.47s (#23)	309.10s (#19)	131.28s (#19)	79.05s (#15)	128.72s (#22)	216.84s (#25)	45.22s (#13)	549.89s (#24)	386.63s (#24)	302.12s (#23)	85.90s (#17)
nemotron-nano-9b-v2	105.27s (#12)	312.82s (#20)	32.68s (#4)	17.92s (#2)	43.74s (#10)	131.37s (#17)	21.33s (#3)	170.01s (#12)	178.13s (#18)	119.10s (#17)	25.57s (#2)
gpt-5-nano	207.20s (#20)	313.96s (#21)	224.05s (#26)	107.75s (#20)	129.28s (#23)	215.35s (#24)	105.33s (#20)	259.00s (#21)	235.53s (#21)	286.03s (#22)	195.73s (#21)
claude-sonnet-4.5	186.47s (#19)	339.56s (#22)	180.11s (#21)	229.01s (#27)	87.05s (#19)	190.26s (#20)	125.27s (#23)	225.01s (#17)	133.99s (#12)	98.12s (#16)	256.35s (#24)
gemini-2.5-pro	169.70s (#18)	414.00s (#23)	97.48s (#14)	87.59s (#18)	52.13s (#14)	115.83s (#14)	61.58s (#15)	189.12s (#14)	161.11s (#17)	409.10s (#24)	109.02s (#18)
Qwen3-235B-A22B-2507	222.55s (#22)	450.74s (#24)	252.51s (#29)	337.09s (#30)	82.08s (#18)	220.56s (#26)	62.45s (#16)	231.14s (#18)	332.28s (#23)	196.04s (#18)	60.62s (#13)
GLM-4.6	361.84s (#26)	451.62s (#25)	198.54s (#25)	119.35s (#21)	149.13s (#25)	404.30s (#31)	121.19s (#22)	645.36s (#29)	800.42s (#32)	466.66s (#29)	261.89s (#26)
claude-opus-4-1	361.89s (#27)	591.22s (#26)	247.29s (#27)	224.17s (#26)	292.51s (#30)	193.13s (#22)	132.67s (#24)	584.89s (#28)	700.23s (#30)	514.58s (#30)	138.21s (#19)
gpt-5.1	385.94s (#29)	626.91s (#27)	248.20s (#28)	321.45s (#29)	197.73s (#28)	321.11s (#29)	258.37s (#30)	575.37s (#26)	583.15s (#26)	434.79s (#26)	292.29s (#29)
gpt-5	434.09s (#32)	653.90s (#28)	308.81s (#30)	429.84s (#32)	271.89s (#29)	403.36s (#30)	259.63s (#31)	650.33s (#30)	420.80s (#25)	535.20s (#32)	407.17s (#32)
deepSeek-R1-0528	347.58s (#25)	696.46s (#29)	118.00s (#17)	203.36s (#24)	126.80s (#21)	191.32s (#21)	174.14s (#27)	777.40s (#31)	618.70s (#27)	210.89s (#20)	358.70s (#31)
Qwen3-235B-A22B-Thinking-2507	420.72s (#30)	712.49s (#30)	187.46s (#24)	223.14s (#25)	195.86s (#27)	220.91s (#27)	177.22s (#28)	900.75s (#32)	778.15s (#31)	532.40s (#31)	278.85s (#27)
Kimi-K2-thinking	428.50s (#31)	734.02s (#31)	512.64s (#32)	402.29s (#31)	164.35s (#26)	286.90s (#28)	208.54s (#29)	575.24s (#25)	696.85s (#29)	451.55s (#27)	252.65s (#23)
deepSeek-v3.2-exp	383.58s (#28)	788.60s (#32)	181.35s (#22)	249.17s (#28)	300.75s (#31)	407.08s (#32)	137.42s (#26)	581.99s (#27)	693.02s (#28)	238.89s (#21)	257.49s (#25)