AutoBench Run 6 - August 2026

Twenty models scored on 252 questions across ten subject domains. Score, cost, and latency are the per-answer committee rank, billed cost, and generation time. Routers in the AutoBench Router Benchmark are scored against this answer matrix.

Latest

Date

August 29, 2026

Version

2026-08-29

Models

New Models

Run data

Overall Comparison Costs Latency P99 Latency Domain

Average latency breakdown across 10 specialized domains, measured in seconds. These metrics help you identify models that consistently deliver quick responses for your specific use case while maintaining quality standards.

Model	Average (All Topics)	Computer Science	Philosophy	Social Science	Language	Science	Technology	Arts	Literature	History	Agentic
Gemma 4 31B IT	15.89s (#1)	29.76s (#2)	8.75s (#1)	17.08s (#3)	13.16s (#1)	27.07s (#1)	18.45s (#1)	17.17s (#3)	14.35s (#3)	11.54s (#3)	3.13s (#1)
Gemini 3.7 Flash	21.56s (#4)	40.61s (#4)	16.87s (#4)	20.71s (#4)	21.16s (#3)	27.47s (#2)	27.44s (#5)	27.79s (#6)	16.56s (#5)	15.89s (#4)	6.09s (#2)
Muse Glimmer 30B	17.02s (#2)	23.97s (#1)	15.27s (#3)	15.83s (#2)	19.18s (#2)	28.57s (#3)	19.91s (#2)	15.65s (#2)	11.57s (#1)	11.51s (#2)	8.75s (#3)
Grok 4.3	19.37s (#3)	36.26s (#3)	12.26s (#2)	14.99s (#1)	21.19s (#4)	38.89s (#4)	19.93s (#3)	14.72s (#1)	14.10s (#2)	10.63s (#1)	9.25s (#5)
Nemotron 3 Ultra	38.07s (#8)	85.53s (#9)	32.16s (#11)	32.24s (#7)	54.21s (#12)	39.80s (#5)	58.82s (#14)	44.18s (#11)	18.58s (#6)	17.26s (#6)	11.54s (#12)
Gpt oss 120b	32.29s (#6)	47.02s (#6)	29.06s (#8)	42.45s (#11)	31.53s (#7)	40.14s (#6)	43.00s (#8)	38.71s (#9)	29.99s (#10)	21.01s (#8)	9.72s (#6)
Hy3	26.58s (#5)	46.53s (#5)	24.12s (#6)	25.19s (#5)	27.94s (#5)	41.42s (#7)	32.97s (#6)	27.54s (#5)	15.57s (#4)	16.91s (#5)	10.65s (#9)
MiniMax M3	44.20s (#10)	93.03s (#11)	50.94s (#17)	42.12s (#10)	40.56s (#9)	44.14s (#8)	48.83s (#11)	70.62s (#16)	33.08s (#12)	32.76s (#16)	10.99s (#10)
Muse Spark 1.2	33.32s (#7)	54.69s (#7)	30.81s (#9)	27.26s (#6)	29.87s (#6)	50.00s (#9)	40.63s (#7)	23.53s (#4)	37.73s (#13)	29.77s (#13)	17.40s (#17)
GLM 5.3	38.26s (#9)	78.66s (#8)	25.37s (#7)	55.66s (#16)	36.62s (#8)	51.44s (#10)	22.60s (#4)	43.49s (#10)	44.15s (#16)	22.74s (#11)	10.11s (#7)
GPT-5.6 Luna	56.52s (#17)	110.46s (#15)	48.75s (#15)	73.73s (#17)	55.36s (#13)	55.20s (#11)	87.72s (#17)	62.84s (#15)	52.06s (#17)	30.99s (#15)	16.39s (#16)
Qwen3.8 Max	54.83s (#16)	87.51s (#10)	50.23s (#16)	51.38s (#15)	74.81s (#16)	63.71s (#12)	65.68s (#16)	85.49s (#18)	39.19s (#14)	30.41s (#14)	14.13s (#15)
Grok 4.6	48.43s (#13)	105.93s (#13)	30.98s (#10)	45.86s (#13)	49.83s (#10)	65.90s (#13)	46.95s (#10)	60.12s (#14)	39.80s (#15)	34.72s (#17)	12.96s (#14)
GPT-5.6 Sol	66.32s (#18)	146.26s (#19)	36.84s (#13)	75.42s (#18)	79.54s (#17)	75.50s (#14)	89.32s (#18)	75.40s (#17)	55.66s (#18)	39.16s (#18)	10.18s (#8)
Kimi K2.6	93.06s (#19)	199.94s (#20)	75.55s (#19)	78.23s (#19)	126.02s (#19)	78.12s (#15)	144.21s (#19)	101.77s (#19)	64.07s (#19)	57.07s (#19)	46.17s (#19)
DeepSeek V4 Flash 0731	53.47s (#15)	124.68s (#17)	51.88s (#18)	49.07s (#14)	64.52s (#15)	82.12s (#16)	58.67s (#13)	52.95s (#12)	29.77s (#9)	26.00s (#12)	12.05s (#13)
Claude Sonnet 5	45.07s (#11)	104.13s (#12)	40.08s (#14)	44.04s (#12)	52.21s (#11)	84.02s (#17)	45.25s (#9)	29.84s (#7)	30.73s (#11)	21.52s (#9)	9.20s (#4)
Nemotron 3.5 Lightning	48.26s (#12)	109.47s (#14)	17.15s (#5)	33.18s (#8)	57.96s (#14)	88.26s (#18)	61.40s (#15)	36.66s (#8)	19.39s (#7)	22.65s (#10)	27.60s (#18)
GLM 5.2	52.38s (#14)	115.30s (#16)	33.16s (#12)	34.78s (#9)	90.25s (#18)	92.81s (#19)	51.13s (#12)	57.43s (#13)	21.51s (#8)	18.61s (#7)	11.39s (#11)
Qwen3.5-9B	144.22s (#20)	131.56s (#18)	138.80s (#20)	131.91s (#20)	202.57s (#20)	183.40s (#20)	176.48s (#20)	149.72s (#20)	137.27s (#20)	122.29s (#20)	92.40s (#20)