AutoBench Run 5 - December 2025

Latest AutoBench run with models Gpt 5.2, Claude Opus 4.5, Gemini 3 Flash and more

Latest

Date

December 19, 2025

Version

2025-12-19

Models

New Models

Run data

Overall Comparison Costs Latency P99 Latency Domain

Average latency breakdown across 10 specialized domains, measured in seconds. These metrics help you identify models that consistently deliver quick responses for your specific use case while maintaining quality standards.

Model	Average (All Topics)	Coding	Creative Writing	Current News	General Culture	Grammar	History	Logics	Math	Science	Technology
gemini-2.5-flash-lite	20.42s (#1)	27.69s (#2)	7.47s (#1)	22.16s (#1)	13.57s (#1)	17.90s (#1)	19.61s (#1)	31.56s (#2)	27.47s (#4)	17.06s (#1)	16.96s (#1)
Nemotron-3-nano-30b-a3b	30.08s (#3)	32.65s (#4)	32.72s (#12)	24.90s (#2)	19.69s (#2)	22.45s (#4)	21.21s (#2)	61.72s (#7)	40.39s (#5)	22.79s (#2)	26.99s (#3)
Gemini-3-flash-preview	45.56s (#6)	53.77s (#7)	32.38s (#11)	26.78s (#3)	22.85s (#3)	25.74s (#5)	21.98s (#3)	151.49s (#22)	80.19s (#13)	24.03s (#4)	25.94s (#2)
grok-4.1-fast	23.60s (#2)	22.59s (#1)	10.52s (#2)	30.52s (#4)	29.92s (#5)	22.21s (#3)	27.63s (#4)	22.90s (#1)	14.23s (#1)	23.01s (#3)	30.30s (#4)
Gpt-oss-20b	38.77s (#5)	48.24s (#6)	14.60s (#3)	37.91s (#5)	32.82s (#6)	38.56s (#6)	29.79s (#6)	64.65s (#8)	47.49s (#6)	31.65s (#5)	40.13s (#6)
Ministral-8b-2512	31.40s (#4)	28.89s (#3)	22.10s (#6)	41.26s (#6)	28.73s (#4)	21.60s (#2)	29.61s (#5)	54.54s (#6)	24.61s (#3)	32.07s (#6)	33.78s (#5)
nemotron-nano-9b-v2	66.78s (#11)	124.96s (#21)	41.48s (#17)	47.07s (#7)	36.69s (#7)	47.29s (#10)	41.35s (#7)	127.95s (#15)	107.37s (#20)	40.87s (#7)	55.15s (#8)
grok-4.1-fast-thinking	69.24s (#12)	99.81s (#13)	27.59s (#7)	50.98s (#8)	55.06s (#9)	58.78s (#15)	53.50s (#9)	147.27s (#19)	93.10s (#16)	58.38s (#13)	58.09s (#9)
llama-3.3-nemotron-super-49b-v1.5	76.48s (#15)	115.78s (#19)	45.77s (#18)	54.40s (#9)	51.81s (#8)	44.54s (#8)	58.91s (#12)	150.50s (#21)	135.10s (#24)	57.97s (#12)	50.26s (#7)
nova-premier-v1	51.84s (#7)	59.24s (#8)	19.71s (#5)	61.11s (#10)	61.19s (#11)	38.99s (#7)	55.96s (#11)	49.03s (#4)	50.09s (#8)	49.47s (#9)	62.25s (#11)
Nova-2-lite-v1	61.46s (#9)	86.86s (#11)	56.83s (#24)	67.08s (#11)	61.45s (#12)	57.90s (#13)	54.84s (#10)	51.97s (#5)	53.26s (#9)	60.47s (#14)	59.86s (#10)
gemini-2.5-flash	65.62s (#10)	122.23s (#20)	28.98s (#9)	68.60s (#12)	58.98s (#10)	57.46s (#12)	53.43s (#8)	77.53s (#10)	70.88s (#12)	47.92s (#8)	65.63s (#13)
gemini-3-pro-preview	76.11s (#14)	111.22s (#17)	41.26s (#16)	68.72s (#13)	67.46s (#13)	63.85s (#16)	59.18s (#13)	138.24s (#17)	94.61s (#18)	57.76s (#11)	63.33s (#12)
Olmo-3.1-32b-think	122.42s (#24)	177.48s (#25)	130.51s (#34)	74.44s (#14)	78.22s (#18)	108.51s (#28)	71.71s (#16)	215.94s (#27)	202.75s (#28)	97.13s (#26)	80.69s (#17)
Qwen3-next-80b-a3b-thinking	77.76s (#16)	105.92s (#14)	48.55s (#19)	77.59s (#15)	69.34s (#14)	85.09s (#19)	65.80s (#14)	98.05s (#11)	85.36s (#15)	71.01s (#17)	68.61s (#14)
Qwen3-235B-A22B-2507	104.78s (#22)	133.33s (#23)	35.13s (#14)	78.98s (#16)	71.22s (#16)	115.15s (#29)	81.85s (#20)	163.34s (#23)	184.08s (#27)	89.48s (#22)	78.70s (#16)
Mistral-medium-3.1	52.25s (#8)	37.22s (#5)	15.67s (#4)	81.62s (#17)	70.42s (#15)	46.77s (#9)	67.96s (#15)	43.20s (#3)	23.28s (#2)	55.35s (#10)	69.74s (#15)
Kimi-K2-0905	82.80s (#17)	96.04s (#12)	49.85s (#22)	87.00s (#18)	91.08s (#22)	57.99s (#14)	103.55s (#24)	129.45s (#16)	56.88s (#11)	67.21s (#16)	94.36s (#22)
gpt-5-nano	99.62s (#21)	111.28s (#18)	56.96s (#25)	97.44s (#19)	93.68s (#23)	104.45s (#27)	90.76s (#22)	149.92s (#20)	109.45s (#21)	89.69s (#23)	94.05s (#21)
gemini-2.5-pro	86.80s (#18)	109.99s (#16)	37.54s (#15)	99.57s (#20)	87.05s (#19)	65.15s (#17)	74.77s (#17)	125.07s (#14)	98.05s (#19)	75.34s (#18)	88.03s (#20)
gpt-oss-120b	75.48s (#13)	85.57s (#10)	30.21s (#10)	107.38s (#21)	77.74s (#17)	54.72s (#11)	80.89s (#19)	119.53s (#13)	48.96s (#7)	66.82s (#15)	85.84s (#19)
Minimax-m2	136.96s (#27)	258.39s (#31)	80.02s (#30)	108.29s (#22)	89.65s (#21)	87.11s (#21)	80.63s (#18)	210.72s (#25)	236.32s (#30)	116.36s (#28)	82.62s (#18)
gpt-5-mini	93.49s (#20)	130.27s (#22)	49.21s (#20)	109.13s (#23)	89.20s (#20)	88.58s (#22)	85.63s (#21)	103.34s (#12)	82.39s (#14)	86.73s (#21)	102.72s (#23)
deepSeek-R1-0528	171.50s (#31)	301.05s (#33)	49.49s (#21)	117.82s (#24)	101.41s (#24)	88.70s (#23)	100.73s (#23)	306.87s (#32)	351.14s (#35)	122.31s (#30)	142.47s (#29)
Mistral-large-2512	89.96s (#19)	79.29s (#9)	28.51s (#8)	126.28s (#25)	111.76s (#27)	80.96s (#18)	115.38s (#25)	70.50s (#9)	54.86s (#10)	93.78s (#25)	116.97s (#26)
Deepseek-v3.2	124.57s (#25)	170.32s (#24)	50.12s (#23)	130.16s (#26)	103.72s (#25)	86.24s (#20)	130.67s (#30)	219.40s (#28)	157.63s (#26)	86.19s (#20)	108.45s (#24)
GLM-4.5-Air	163.15s (#29)	250.89s (#30)	33.18s (#13)	130.32s (#27)	118.61s (#28)	102.73s (#25)	125.26s (#29)	275.60s (#31)	337.88s (#34)	103.09s (#27)	122.18s (#27)
claude-haiku-4.5	110.95s (#23)	108.22s (#15)	77.98s (#29)	139.82s (#28)	119.32s (#29)	96.88s (#24)	118.54s (#26)	146.18s (#18)	93.19s (#17)	85.52s (#19)	128.07s (#28)
grok-4	180.11s (#32)	298.51s (#32)	65.54s (#27)	144.69s (#29)	119.88s (#30)	154.57s (#33)	121.90s (#28)	391.60s (#33)	277.86s (#31)	135.10s (#32)	151.08s (#30)
Gpt-5.2	130.10s (#26)	215.39s (#29)	72.54s (#28)	154.21s (#30)	108.34s (#26)	115.26s (#30)	120.99s (#27)	211.13s (#26)	111.21s (#23)	91.37s (#24)	111.20s (#25)
Claude-opus-4.5	144.01s (#28)	205.12s (#28)	91.87s (#31)	160.29s (#31)	154.82s (#31)	103.08s (#26)	161.93s (#32)	168.86s (#24)	110.98s (#22)	119.37s (#29)	160.87s (#32)
Kimi-K2-thinking	247.97s (#35)	357.90s (#35)	211.44s (#37)	187.67s (#32)	174.58s (#33)	156.16s (#34)	192.82s (#34)	406.50s (#35)	400.05s (#37)	189.54s (#36)	217.88s (#37)
GLM-4.6	187.43s (#33)	187.91s (#26)	59.68s (#26)	192.02s (#33)	172.91s (#32)	143.98s (#32)	161.75s (#31)	273.65s (#30)	325.95s (#33)	153.79s (#33)	160.86s (#31)
claude-sonnet-4.5	169.73s (#30)	194.27s (#27)	97.99s (#32)	221.78s (#34)	197.14s (#35)	123.52s (#31)	172.17s (#33)	224.11s (#29)	153.14s (#25)	126.51s (#31)	185.53s (#33)
Gpt-5.2-pro	261.38s (#36)	391.12s (#36)	208.12s (#36)	226.37s (#35)	193.88s (#34)	223.70s (#36)	204.98s (#36)	576.64s (#36)	317.31s (#32)	156.86s (#34)	194.85s (#35)
Deepseek-v3.2-speciale	310.39s (#37)	454.41s (#37)	373.53s (#38)	230.84s (#36)	213.12s (#37)	265.55s (#38)	201.24s (#35)	651.16s (#38)	388.44s (#36)	216.22s (#37)	251.44s (#38)
gpt-5.1	227.43s (#34)	311.14s (#34)	145.02s (#35)	239.83s (#37)	201.64s (#36)	180.41s (#35)	222.08s (#37)	400.03s (#34)	232.09s (#29)	177.33s (#35)	190.64s (#34)
Qwen3-235B-A22B-Thinking-2507	316.82s (#38)	548.95s (#38)	102.39s (#33)	246.60s (#38)	235.78s (#38)	240.61s (#37)	257.27s (#38)	631.62s (#37)	589.95s (#38)	236.47s (#38)	208.52s (#36)