AutoBench Run 5 - December 2025

Latest AutoBench run with models Gpt 5.2, Claude Opus 4.5, Gemini 3 Flash and more

Latest

Date

December 19, 2025

Version

2025-12-19

Models

New Models

Run data

Overall Comparison Costs Latency P99 Latency Domain

P99 latency breakdown across 10 specialized domains, measured in seconds. This 99th percentile metric reveals worst-case response times, helping you ensure consistent performance for production environments.

Model	Average (All Topics)	Coding	Creative Writing	Current News	General Culture	Grammar	History	Logics	Math	Science	Technology
Deepseek-v3.2-speciale	832.67s (#38)	1311.96s (#36)	650.23s (#38)	528.49s (#37)	503.19s (#32)	607.86s (#38)	403.12s (#32)	1755.13s (#38)	1282.71s (#34)	701.66s (#38)	582.35s (#38)
Qwen3-235B-A22B-Thinking-2507	810.89s (#37)	1366.04s (#37)	248.13s (#29)	506.33s (#36)	540.14s (#34)	515.38s (#35)	580.07s (#37)	1677.41s (#36)	1660.09s (#38)	582.45s (#37)	432.82s (#35)
deepSeek-R1-0528	476.65s (#30)	844.76s (#28)	99.46s (#15)	344.66s (#27)	310.40s (#27)	362.74s (#27)	220.56s (#24)	737.76s (#30)	1043.45s (#31)	548.97s (#36)	253.75s (#25)
GLM-4.6	630.49s (#34)	726.63s (#26)	152.22s (#18)	537.89s (#38)	591.02s (#38)	502.27s (#33)	618.98s (#38)	898.32s (#32)	1317.98s (#36)	546.11s (#35)	413.46s (#34)
Kimi-K2-thinking	729.12s (#35)	1063.41s (#35)	599.81s (#37)	401.07s (#30)	574.37s (#36)	506.63s (#34)	437.62s (#34)	1223.91s (#34)	1507.26s (#37)	520.56s (#34)	456.51s (#36)
gpt-5.1	627.33s (#33)	991.04s (#32)	369.43s (#34)	416.66s (#31)	430.18s (#31)	560.75s (#36)	468.29s (#35)	1079.65s (#33)	1077.59s (#33)	512.74s (#33)	366.99s (#32)
Gpt-5.2-pro	783.82s (#36)	1424.65s (#38)	435.14s (#35)	477.57s (#34)	586.24s (#37)	582.91s (#37)	431.57s (#33)	1706.35s (#37)	1289.05s (#35)	425.52s (#32)	479.20s (#37)
gpt-oss-120b	291.84s (#21)	309.68s (#14)	98.59s (#12)	308.84s (#24)	258.63s (#24)	313.48s (#26)	194.04s (#20)	684.89s (#29)	180.43s (#10)	380.11s (#31)	189.75s (#13)
grok-4	562.46s (#32)	1010.63s (#34)	281.41s (#33)	304.53s (#23)	211.69s (#18)	448.43s (#32)	189.40s (#19)	1420.30s (#35)	1052.80s (#32)	377.39s (#30)	327.97s (#30)
Minimax-m2	472.81s (#29)	991.82s (#33)	589.49s (#36)	238.88s (#16)	288.48s (#25)	400.58s (#30)	201.58s (#22)	562.56s (#27)	746.35s (#29)	374.69s (#29)	333.69s (#31)
claude-sonnet-4.5	476.75s (#31)	847.62s (#29)	216.36s (#26)	502.25s (#35)	555.51s (#35)	406.21s (#31)	564.10s (#36)	526.61s (#23)	428.48s (#23)	340.18s (#28)	380.19s (#33)
Olmo-3.1-32b-think	270.44s (#20)	345.25s (#17)	273.29s (#32)	135.28s (#7)	136.91s (#9)	222.38s (#16)	107.56s (#9)	467.38s (#17)	507.04s (#25)	302.23s (#27)	207.07s (#17)
Qwen3-235B-A22B-2507	337.21s (#24)	388.67s (#19)	256.53s (#30)	274.04s (#20)	224.98s (#20)	386.17s (#28)	290.78s (#28)	524.49s (#22)	507.05s (#26)	284.57s (#26)	234.87s (#24)
Gpt-5.2	434.15s (#28)	983.23s (#31)	173.15s (#22)	458.68s (#33)	401.14s (#30)	278.51s (#24)	244.60s (#26)	743.59s (#31)	574.40s (#27)	283.53s (#25)	200.63s (#15)
Claude-opus-4.5	373.31s (#25)	834.12s (#27)	210.10s (#25)	329.94s (#26)	377.65s (#29)	254.76s (#20)	382.09s (#30)	493.52s (#20)	264.77s (#15)	281.70s (#24)	304.43s (#28)
Qwen3-next-80b-a3b-thinking	226.68s (#16)	497.62s (#22)	107.44s (#17)	187.07s (#12)	154.37s (#13)	247.60s (#19)	120.27s (#13)	189.55s (#9)	343.78s (#19)	279.02s (#23)	140.04s (#9)
Deepseek-v3.2	410.46s (#26)	631.21s (#25)	193.29s (#23)	356.02s (#28)	313.10s (#28)	393.95s (#29)	395.74s (#31)	541.90s (#25)	691.56s (#28)	278.37s (#22)	309.50s (#29)
GLM-4.5-Air	425.28s (#27)	945.31s (#30)	88.57s (#11)	379.63s (#29)	236.09s (#23)	270.87s (#23)	253.79s (#27)	632.83s (#28)	900.50s (#30)	274.40s (#21)	270.86s (#26)
gpt-5-mini	257.83s (#18)	560.12s (#24)	166.98s (#20)	239.95s (#18)	185.68s (#15)	267.72s (#22)	200.70s (#21)	246.31s (#10)	214.93s (#11)	269.22s (#20)	226.66s (#23)
gpt-5-nano	268.83s (#19)	282.18s (#10)	98.87s (#14)	238.91s (#17)	231.81s (#22)	302.92s (#25)	185.82s (#18)	490.26s (#19)	440.47s (#24)	259.70s (#19)	157.38s (#11)
llama-3.3-nemotron-super-49b-v1.5	240.32s (#17)	408.06s (#20)	271.44s (#31)	160.54s (#9)	133.01s (#8)	155.29s (#8)	142.28s (#16)	299.67s (#12)	423.42s (#22)	257.52s (#18)	151.94s (#10)
claude-haiku-4.5	316.92s (#22)	303.69s (#12)	172.93s (#21)	283.97s (#22)	514.83s (#33)	246.10s (#18)	325.97s (#29)	418.19s (#15)	361.13s (#21)	252.93s (#17)	289.47s (#27)
grok-4.1-fast-thinking	207.02s (#13)	285.18s (#11)	73.44s (#9)	100.46s (#4)	138.18s (#10)	177.54s (#11)	99.14s (#6)	516.52s (#21)	266.01s (#16)	246.66s (#16)	167.10s (#12)
gemini-2.5-pro	222.47s (#15)	306.23s (#13)	63.38s (#7)	314.20s (#25)	209.90s (#17)	160.49s (#10)	139.17s (#15)	328.10s (#13)	268.02s (#17)	228.24s (#15)	207.00s (#16)
Mistral-large-2512	198.13s (#12)	318.52s (#15)	67.47s (#8)	277.47s (#21)	230.39s (#21)	182.53s (#15)	212.87s (#23)	132.59s (#6)	141.92s (#7)	204.26s (#14)	213.26s (#19)
Kimi-K2-0905	329.01s (#23)	537.40s (#23)	227.50s (#27)	434.36s (#32)	300.18s (#26)	262.83s (#21)	233.52s (#25)	537.19s (#24)	355.69s (#20)	188.43s (#13)	212.96s (#18)
nemotron-nano-9b-v2	211.73s (#14)	338.66s (#16)	195.02s (#24)	111.18s (#5)	64.85s (#4)	178.36s (#13)	92.15s (#4)	473.75s (#18)	272.36s (#18)	169.01s (#12)	222.00s (#21)
Mistral-medium-3.1	146.93s (#7)	101.37s (#4)	25.43s (#3)	207.95s (#13)	212.72s (#19)	182.36s (#14)	175.82s (#17)	108.12s (#4)	63.58s (#1)	166.40s (#11)	225.52s (#22)
Gpt-oss-20b	183.02s (#10)	279.10s (#9)	37.39s (#5)	221.63s (#14)	94.44s (#7)	235.04s (#17)	117.91s (#12)	253.53s (#11)	225.41s (#12)	147.66s (#10)	218.10s (#20)
nova-premier-v1	134.54s (#5)	177.99s (#8)	35.03s (#4)	225.56s (#15)	188.51s (#16)	93.04s (#5)	103.82s (#7)	83.18s (#2)	94.83s (#5)	145.68s (#9)	197.75s (#14)
Nova-2-lite-v1	131.60s (#4)	173.45s (#7)	103.96s (#16)	157.20s (#8)	159.48s (#14)	133.87s (#7)	108.75s (#10)	116.05s (#5)	119.65s (#6)	135.61s (#8)	107.99s (#6)
gemini-3-pro-preview	186.15s (#11)	381.70s (#18)	98.86s (#13)	168.30s (#10)	140.84s (#12)	157.04s (#9)	98.37s (#5)	342.94s (#14)	247.19s (#13)	129.19s (#7)	97.11s (#5)
gemini-2.5-flash	173.94s (#9)	475.33s (#21)	58.33s (#6)	184.80s (#11)	139.99s (#11)	178.27s (#12)	125.50s (#14)	154.38s (#7)	178.77s (#9)	112.88s (#6)	131.14s (#8)
Nemotron-3-nano-30b-a3b	97.87s (#3)	71.82s (#2)	159.59s (#19)	48.94s (#1)	33.30s (#1)	107.56s (#6)	42.97s (#2)	177.95s (#8)	172.55s (#8)	110.82s (#5)	53.20s (#3)
Ministral-8b-2512	154.14s (#8)	108.05s (#5)	233.48s (#28)	256.42s (#19)	82.40s (#6)	55.00s (#3)	104.63s (#8)	423.65s (#16)	68.72s (#2)	92.10s (#4)	116.93s (#7)
gemini-2.5-flash-lite	69.09s (#2)	78.70s (#3)	11.68s (#1)	128.71s (#6)	38.80s (#3)	44.97s (#1)	110.24s (#11)	86.64s (#3)	74.84s (#4)	84.66s (#3)	31.65s (#1)
Gemini-3-flash-preview	136.57s (#6)	151.27s (#6)	87.09s (#10)	51.96s (#2)	38.37s (#2)	70.02s (#4)	37.09s (#1)	549.69s (#26)	257.35s (#14)	84.60s (#2)	38.28s (#2)
grok-4.1-fast	57.44s (#1)	49.75s (#1)	20.18s (#2)	54.43s (#3)	78.25s (#5)	46.27s (#2)	49.49s (#3)	77.61s (#1)	71.87s (#3)	48.17s (#1)	78.36s (#4)