AutoBench Run 4 - November 2025

Latest AutoBench run with models Gemini 3 Pro, Gpt 5.1, Grok 4.1 and more

Past

Date

November 28, 2025

Version

2025-11-28

Models

New Models

Run data

Overall Comparison Costs Latency P99 Latency Domain

Average latency breakdown across 10 specialized domains, measured in seconds. These metrics help you identify models that consistently deliver quick responses for your specific use case while maintaining quality standards.

Model	Average (All Topics)	Coding	Creative Writing	Current News	General Culture	Grammar	History	Logics	Math	Science	Technology
nova-pro-v1	6.61s (#1)	9.31s (#1)	8.30s (#1)	7.11s (#1)	6.40s (#1)	6.03s (#1)	5.62s (#1)	4.76s (#1)	4.17s (#1)	5.39s (#1)	8.98s (#2)
magistral-medium-2506	17.43s (#7)	22.33s (#5)	10.48s (#4)	14.43s (#7)	8.23s (#2)	10.11s (#2)	8.73s (#2)	41.66s (#11)	40.95s (#13)	8.78s (#2)	8.63s (#1)
llama-4-maverick	12.94s (#2)	17.59s (#2)	10.18s (#2)	10.65s (#2)	10.88s (#4)	16.79s (#9)	10.61s (#3)	14.42s (#3)	17.78s (#6)	11.13s (#3)	9.35s (#3)
gemini-2.5-flash-lite	14.20s (#4)	24.36s (#6)	10.26s (#3)	11.32s (#4)	13.27s (#7)	12.91s (#4)	10.84s (#4)	21.00s (#6)	14.94s (#5)	12.54s (#4)	10.54s (#5)
nova-premier-v1	13.06s (#3)	20.29s (#3)	13.44s (#7)	12.32s (#5)	12.03s (#6)	11.81s (#3)	14.13s (#7)	11.36s (#2)	10.18s (#2)	13.82s (#5)	11.19s (#7)
mistral-small-3.2-24b-instruct	14.54s (#5)	21.76s (#4)	11.19s (#5)	13.11s (#6)	8.96s (#3)	14.88s (#6)	11.24s (#5)	17.13s (#5)	20.98s (#8)	13.85s (#6)	12.35s (#9)
llama-3.3-70B-Instruct	18.31s (#9)	27.13s (#8)	17.83s (#11)	15.83s (#8)	16.76s (#12)	15.24s (#7)	21.67s (#13)	21.25s (#7)	22.07s (#9)	14.44s (#7)	10.86s (#6)
gpt-oss-120b	16.97s (#6)	27.43s (#9)	14.35s (#8)	19.73s (#12)	14.21s (#10)	14.35s (#5)	17.03s (#10)	22.23s (#8)	13.22s (#3)	15.65s (#8)	11.47s (#8)
grok-3-mini	19.74s (#10)	31.28s (#10)	16.33s (#9)	17.41s (#10)	14.10s (#9)	20.14s (#11)	14.21s (#8)	24.79s (#9)	27.07s (#10)	15.82s (#9)	16.24s (#11)
nemotron-nano-9b-v2	24.53s (#12)	55.45s (#13)	12.84s (#6)	10.94s (#3)	11.41s (#5)	19.71s (#10)	11.82s (#6)	52.14s (#14)	45.46s (#15)	15.99s (#10)	9.54s (#4)
phi-4	17.78s (#8)	26.36s (#7)	19.75s (#12)	17.23s (#9)	14.01s (#8)	15.30s (#8)	15.86s (#9)	15.16s (#4)	20.17s (#7)	17.26s (#11)	16.72s (#12)
Qwen3-30B-A3B-Instruct-2507	37.56s (#14)	65.33s (#16)	20.50s (#13)	18.16s (#11)	15.87s (#11)	28.07s (#15)	19.58s (#11)	93.98s (#22)	72.20s (#20)	22.09s (#12)	19.79s (#13)
grok-4.1-fast	23.12s (#11)	33.26s (#11)	16.38s (#10)	24.15s (#13)	18.56s (#13)	22.10s (#13)	23.04s (#14)	34.62s (#10)	14.93s (#4)	23.15s (#13)	21.02s (#14)
gemma-3-27b-it	36.12s (#13)	44.77s (#12)	50.42s (#23)	25.84s (#15)	45.47s (#23)	25.04s (#14)	32.65s (#19)	45.77s (#13)	33.29s (#11)	23.16s (#14)	34.80s (#19)
llama-3.3-nemotron-super-49b-v1.5	43.88s (#18)	80.69s (#20)	26.76s (#14)	24.39s (#14)	23.73s (#15)	42.35s (#17)	24.00s (#16)	94.50s (#23)	73.99s (#21)	26.54s (#15)	21.87s (#15)
Qwen3-235B-A22B-2507	43.08s (#17)	74.27s (#19)	29.16s (#17)	37.78s (#20)	23.75s (#16)	43.28s (#18)	20.57s (#12)	78.39s (#20)	80.44s (#23)	27.64s (#16)	15.51s (#10)
claude-haiku-4.5	41.70s (#16)	58.74s (#14)	45.16s (#21)	33.85s (#19)	25.82s (#17)	49.31s (#21)	33.47s (#20)	56.56s (#15)	39.31s (#12)	29.81s (#17)	45.02s (#21)
gemini-3-pro-preview	45.51s (#20)	63.68s (#15)	44.61s (#20)	33.22s (#18)	30.14s (#18)	48.30s (#20)	32.38s (#18)	67.64s (#17)	63.04s (#18)	40.60s (#18)	31.45s (#17)
deepSeek-v3.2-exp	99.52s (#26)	169.07s (#29)	55.13s (#25)	56.81s (#25)	56.61s (#27)	78.93s (#24)	48.64s (#23)	215.16s (#29)	219.92s (#30)	43.04s (#19)	51.91s (#23)
claude-sonnet-4.5	57.72s (#22)	89.25s (#21)	54.27s (#24)	64.14s (#28)	42.13s (#22)	54.53s (#22)	53.51s (#24)	70.52s (#18)	52.31s (#17)	43.10s (#20)	53.43s (#25)
gemini-2.5-flash	39.32s (#15)	68.06s (#17)	27.70s (#15)	27.57s (#16)	22.32s (#14)	33.64s (#16)	23.29s (#15)	59.96s (#16)	52.23s (#16)	47.32s (#21)	31.15s (#16)
deepSeek-R1-0528	106.80s (#28)	222.85s (#31)	45.31s (#22)	63.75s (#27)	48.20s (#24)	57.56s (#23)	61.37s (#27)	231.83s (#31)	225.24s (#31)	50.58s (#22)	61.30s (#27)
grok-4.1-fast-thinking	63.12s (#23)	98.02s (#24)	41.57s (#19)	32.43s (#17)	30.69s (#19)	81.13s (#26)	30.15s (#17)	134.14s (#24)	96.45s (#24)	55.05s (#23)	31.54s (#18)
gemini-2.5-pro	53.73s (#21)	91.24s (#23)	38.17s (#18)	41.12s (#22)	38.69s (#20)	47.20s (#19)	41.90s (#22)	72.09s (#19)	66.61s (#19)	56.27s (#24)	44.06s (#20)
Kimi-K2-0905	44.61s (#19)	72.38s (#18)	28.80s (#16)	39.44s (#21)	51.22s (#25)	20.73s (#12)	39.67s (#21)	44.97s (#12)	41.35s (#14)	58.04s (#25)	49.51s (#22)
gpt-5-nano	68.36s (#24)	91.12s (#22)	69.75s (#27)	53.53s (#24)	51.30s (#26)	84.19s (#28)	53.88s (#25)	91.67s (#21)	77.48s (#22)	58.48s (#26)	52.22s (#24)
Kimi-K2-thinking	85.59s (#25)	126.17s (#25)	105.89s (#31)	47.86s (#23)	41.59s (#21)	79.19s (#25)	55.13s (#26)	174.72s (#25)	107.27s (#25)	63.48s (#27)	54.61s (#26)
GLM-4.6	99.87s (#27)	133.17s (#26)	68.04s (#26)	61.79s (#26)	69.63s (#29)	98.10s (#30)	66.16s (#28)	184.19s (#26)	154.26s (#27)	88.91s (#28)	74.49s (#29)
claude-opus-4-1	113.53s (#29)	160.94s (#28)	84.62s (#29)	90.40s (#30)	92.83s (#31)	81.91s (#27)	85.48s (#30)	193.87s (#27)	174.72s (#29)	96.42s (#29)	74.05s (#28)
Qwen3-235B-A22B-Thinking-2507	138.21s (#31)	231.32s (#32)	74.95s (#28)	88.99s (#29)	68.23s (#28)	85.99s (#29)	84.89s (#29)	287.21s (#32)	264.19s (#32)	115.29s (#30)	81.03s (#30)
gpt-5.1	129.24s (#30)	158.93s (#27)	101.48s (#30)	119.54s (#31)	88.99s (#30)	122.30s (#31)	137.47s (#31)	194.24s (#28)	158.55s (#28)	120.79s (#31)	90.07s (#31)
gpt-5	151.05s (#32)	196.51s (#30)	123.35s (#32)	162.45s (#32)	120.26s (#32)	140.44s (#32)	153.60s (#32)	219.72s (#30)	123.24s (#26)	145.39s (#32)	125.56s (#32)