AutoBench Run 6 - August 2026

Twenty models scored on 252 questions across ten subject domains. Score, cost, and latency are the per-answer committee rank, billed cost, and generation time. Routers in the AutoBench Router Benchmark are scored against this answer matrix.

Latest

Date

August 29, 2026

Version

2026-08-29

Models

New Models

Run data

Overall Comparison Costs Latency P99 Latency Domain

P99 latency breakdown across 10 specialized domains, measured in seconds. This 99th percentile metric reveals worst-case response times, helping you ensure consistent performance for production environments.

Model	Average (All Topics)	Computer Science	Philosophy	Social Science	Language	Science	Technology	Arts	Literature	History	Agentic
Qwen3.5-9B	488.25s (#20)	285.18s (#8)	262.10s (#20)	332.57s (#19)	509.58s (#19)	555.65s (#20)	469.04s (#20)	355.05s (#16)	328.50s (#19)	362.34s (#20)	317.22s (#20)
Kimi K2.6	487.51s (#19)	514.94s (#18)	185.84s (#18)	210.49s (#15)	559.35s (#20)	268.77s (#10)	378.15s (#18)	361.90s (#17)	167.32s (#16)	153.11s (#18)	171.78s (#19)
Nemotron 3.5 Lightning	343.88s (#12)	335.60s (#11)	47.99s (#5)	118.81s (#6)	279.25s (#12)	386.92s (#17)	304.95s (#17)	224.54s (#10)	37.27s (#4)	124.86s (#15)	136.10s (#18)
Nemotron 3 Ultra	322.09s (#11)	322.53s (#9)	148.41s (#14)	130.90s (#10)	354.53s (#14)	255.64s (#8)	453.09s (#19)	273.67s (#13)	60.63s (#6)	45.38s (#5)	97.26s (#17)
GLM 5.2	472.50s (#18)	556.74s (#20)	87.88s (#10)	156.53s (#12)	464.45s (#18)	405.37s (#19)	168.32s (#9)	488.91s (#20)	73.26s (#8)	51.12s (#6)	71.60s (#16)
GPT-5.6 Luna	298.19s (#10)	350.95s (#12)	160.19s (#17)	305.56s (#18)	244.04s (#11)	232.04s (#7)	259.32s (#13)	252.52s (#11)	212.75s (#17)	142.50s (#17)	70.10s (#15)
Muse Spark 1.2	193.84s (#7)	328.24s (#10)	102.80s (#11)	124.08s (#8)	117.26s (#7)	256.56s (#9)	172.00s (#10)	58.23s (#3)	103.19s (#11)	114.10s (#13)	68.67s (#14)
GPT-5.6 Sol	400.97s (#16)	427.59s (#15)	158.64s (#16)	446.85s (#20)	369.60s (#16)	364.33s (#15)	270.72s (#16)	277.75s (#14)	340.12s (#20)	160.89s (#19)	58.05s (#13)
DeepSeek V4 Flash 0731	371.82s (#13)	485.43s (#17)	216.75s (#19)	273.77s (#17)	328.31s (#13)	339.48s (#14)	268.12s (#14)	302.65s (#15)	103.09s (#10)	75.03s (#10)	54.44s (#12)
Grok 4.6	237.80s (#8)	516.02s (#19)	63.22s (#6)	173.83s (#14)	167.89s (#8)	394.91s (#18)	124.42s (#7)	216.79s (#9)	103.35s (#12)	86.53s (#12)	44.64s (#11)
MiniMax M3	376.61s (#14)	388.77s (#14)	154.50s (#15)	125.17s (#9)	231.23s (#10)	333.05s (#13)	183.63s (#11)	422.19s (#18)	110.44s (#14)	140.82s (#16)	32.41s (#10)
Qwen3.8 Max	408.55s (#17)	256.71s (#7)	147.18s (#13)	158.09s (#13)	404.50s (#17)	300.07s (#12)	269.86s (#15)	452.31s (#19)	156.35s (#15)	123.57s (#14)	26.68s (#9)
GLM 5.3	277.53s (#9)	482.28s (#16)	65.53s (#7)	264.70s (#16)	186.98s (#9)	273.80s (#11)	40.75s (#1)	271.20s (#12)	226.03s (#18)	76.93s (#11)	26.59s (#8)
Hy3	112.41s (#2)	90.68s (#2)	72.44s (#9)	56.28s (#3)	85.18s (#5)	127.40s (#4)	102.84s (#5)	70.58s (#4)	33.56s (#3)	42.28s (#4)	25.84s (#7)
Gpt oss 120b	145.77s (#6)	119.32s (#4)	65.65s (#8)	121.16s (#7)	82.84s (#4)	148.78s (#5)	219.60s (#12)	197.87s (#8)	110.27s (#13)	68.68s (#9)	22.48s (#6)
Claude Sonnet 5	381.49s (#15)	381.29s (#13)	104.85s (#12)	150.73s (#11)	360.23s (#15)	382.06s (#16)	140.66s (#8)	95.99s (#5)	92.07s (#9)	60.93s (#8)	21.86s (#5)
Gemini 3.7 Flash	116.93s (#3)	112.00s (#3)	44.49s (#3)	73.25s (#5)	103.81s (#6)	104.53s (#2)	106.85s (#6)	175.84s (#7)	46.79s (#5)	41.89s (#3)	19.77s (#4)
Grok 4.3	142.98s (#5)	139.08s (#5)	21.79s (#2)	32.97s (#2)	77.17s (#3)	190.17s (#6)	63.61s (#4)	34.02s (#1)	32.05s (#2)	21.19s (#1)	18.08s (#3)
Muse Glimmer 30B	82.39s (#1)	79.36s (#1)	47.82s (#4)	27.41s (#1)	51.06s (#2)	84.48s (#1)	59.76s (#3)	37.39s (#2)	24.91s (#1)	28.99s (#2)	15.21s (#2)
Gemma 4 31B IT	117.31s (#4)	150.80s (#6)	19.39s (#1)	72.74s (#4)	38.92s (#1)	117.35s (#3)	43.92s (#2)	128.34s (#6)	71.17s (#7)	54.03s (#7)	11.05s (#1)