Cheap and strong is empty
No graded pass sits in the cheap-and-strong region. The routers that are accurate are not the cheap ones.
The AI Evaluation Engine
for the Agentic Era
The engine benchmarks models, agents, verticals, and now routers. Quality, cost, and speed come from the same evidence.
New Router benchmark is live (7 commercial routers). None beat calling a mid-priced model on both quality and cost.
AutoBench Router Benchmark
7 commercial routers, 17 passes, scored on the 252-question matrix from 2026-08-29. Accuracy is the share of questions where the chosen model is statistically as good as the best available answer.
5 of 17 passes collapsed
A collapsed pass sends at least 90% of questions to one model. It is a constant, not a router.
Same-pick median 0
Across 59 cross-vendor pairs, the median share of questions where two passes name the same model is 0.
Same-pick
Median cross-vendor same-pick is 0 across 59 pairs. The grid is the 12-pass comparison from the report. A cell is the share of questions on which the two passes named the same model.
| Orca cheap | Orca bal | Orca qual | Azure cost | Azure bal | Azure qual | NadirClaw | Brick | Cascade | ND k5 | Auto high | Auto low | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Orca cheap | 1.00 | 0.99 | 0.01 | 0.14 | 0.01 | 0.00 | 0.02 | 0.00 | 0.01 | 0.00 | 0.00 | 0.01 |
| Orca bal | 0.99 | 1.00 | 0.00 | 0.14 | 0.01 | 0.00 | 0.02 | 0.00 | 0.00 | 0.00 | 0.00 | 0.01 |
| Orca qual | 0.01 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.80 | 0.58 | 0.00 | 0.00 | 0.98 |
| Azure cost | 0.14 | 0.14 | 0.00 | 1.00 | 0.19 | 0.03 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| Azure bal | 0.01 | 0.01 | 0.00 | 0.19 | 1.00 | 0.11 | 0.01 | 0.00 | 0.00 | 0.00 | 0.00 | 0.01 |
| Azure qual | 0.00 | 0.00 | 0.00 | 0.03 | 0.11 | 1.00 | 0.12 | 0.00 | 0.00 | 0.00 | 0.01 | 0.00 |
| NadirClaw | 0.02 | 0.02 | 0.00 | 0.00 | 0.01 | 0.12 | 1.00 | 0.00 | 0.00 | 0.34 | 0.00 | 0.00 |
| Brick | 0.00 | 0.00 | 0.80 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.48 | 0.00 | 0.00 | 0.79 |
| Cascade | 0.01 | 0.00 | 0.58 | 0.00 | 0.00 | 0.00 | 0.00 | 0.48 | 1.00 | 0.00 | 0.00 | 0.57 |
| ND k5 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.34 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 |
| Auto high | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.01 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 |
| Auto low | 0.01 | 0.01 | 0.98 | 0.00 | 0.01 | 0.00 | 0.00 | 0.79 | 0.57 | 0.00 | 0.00 | 1.00 |
Routing without measurement is guessing
One engine writes the questions, collects the answers, and scores them. Benchmarks and routing evidence come from that same record.
Live
Benchmarks
Models, agents, and verticals, scored on quality, cost, and speed. Each run writes new questions, so the board is hard to game.
See the leaderboards →Benchmark live
Routing
The Router Benchmark grades commercial routers on a frozen answer matrix. A trained router, built on the same evidence, is next.
See the Router Benchmark →More on the same engine
Your evidence
Private model runs, a router graded on your traffic, or a router built for your own model pool.
Request a custom benchmark →Leaderboards
Average Score combining domain-specific Autobench scores; Higher is better
Benchmarks
Model | Score | Avg Cost ($ Cents) | Avg Latency (sec) | P99 Latency (sec) | Iterations |
|---|---|---|---|---|---|
| 4.78 (#1) | 7.87 (#20) | 66s (#18) | 401s (#16) | 248 | |
| 4.74 (#2) | 1.63 (#17) | 55s (#16) | 409s (#17) | 246 | |
| 4.74 (#3) | 0.43 (#8) | 57s (#17) | 298s (#10) | 251 | |
| 4.73 (#4) | 4.05 (#19) | 45s (#11) | 381s (#15) | 243 | |
| 4.69 (#5) | 1.45 (#16) | 93s (#19) | 488s (#19) | 238 | |
| 4.68 (#6) | 1.07 (#14) | 22s (#4) | 117s (#3) | 251 | |
| 4.67 (#7) | 0.51 (#11) | 52s (#14) | 473s (#18) | 247 | |
| 4.65 (#8) | 0.44 (#9) | 44s (#10) | 377s (#14) | 244 | |
| 4.65 (#9) | 0.57 (#12) | 38s (#9) | 278s (#9) | 243 | |
| 4.61 (#10) | 0.09 (#4) | 27s (#5) | 112s (#2) | 252 | |
| 4.60 (#11) | 1.64 (#18) | 48s (#13) | 238s (#8) | 247 | |
| 4.59 (#12) | 0.05 (#3) | 53s (#15) | 372s (#13) | 245 | |
| 4.56 (#13) | 1.15 (#15) | 33s (#7) | 194s (#7) | 251 | |
| 4.49 (#14) | 0.69 (#13) | 38s (#8) | 322s (#11) | 244 | |
| 4.44 (#15) | 0.50 (#10) | 19s (#3) | 143s (#5) | 252 | |
| 4.36 (#16) | 0.15 (#7) | 48s (#12) | 344s (#12) | 252 | |
| 4.35 (#17) | 0.11 (#6) | 17s (#2) | 82s (#1) | 252 | |
| 4.35 (#18) | 0.02 (#2) | 16s (#1) | 117s (#4) | 252 | |
| 4.15 (#19) | 0.10 (#5) | 144s (#20) | 488s (#20) | 223 | |
| 4.12 (#20) | 0.02 (#1) | 32s (#6) | 146s (#6) | 252 |
HOW IT WORKS
One engine writes the questions, collects the answers, and scores them. Benchmarks and routing evidence are two readings of that record.
01 Generate
The engine writes new, difficulty-balanced questions. There is no fixed public test set.
02 Answer
Each model answers. Every answer carries its billed cost and its generation time.
03 Judge
A committee of models scores every answer. Consensus is the signal.
04 Use the evidence
The same run is a benchmark report, and the trace a router can be graded on.
Validation: Proven Accuracy at Scale
AutoBench’s effectiveness is not theoretical. The results from its public runs demonstrate both unprecedented scale and exceptionally high correlation with industry-standard benchmarks
- 88.13%AAII
- 32ranked models
- 16ranking models
- 320kranks
- 10512iterations
- 336kanswers
- ≈300Mgenerated tokens
- ≈8Bevaluated tokens
AutoBench for Enterprises and LLM Labs
For Enterprises with Large-Scale LLM Consumption
Benchmark models on your use cases
Run the engine on your own tasks and data. You get quality, cost, and latency for the models you actually call, including sentiment analysis, document summarization, and customer support.
Grade routers on your traffic
The public Router Benchmark scores commercial routers on a frozen answer matrix. The same method can grade a router on your own traffic. The latest public model run correlates 88.13% with the Artificial Analysis Intelligence Index.
A router for your mix
When the public board is not the question you need answered, we can build a router on your model pool and your evidence.
Empowering LLM Lab R&D
Benchmark your Use Cases
With over 20 major labs competing, granular evaluation is critical. AutoBench offers private, domain-focused benchmarks to reveal weaknesses in areas like advanced reasoning or specific coding.
See $ Trade-Offs Instantly
Get instant, nuanced views of performance trade-offs through collective LLM judging. The latest public run correlates 88.13% with the Artificial Analysis Intelligence Index, giving labs actionable data to refine models efficiently.
Switch & Monitor
Monitor progress and switch training strategies with ease. Our scalable framework supports continuous custom runs, helping labs adapt architectures and data for better outcomes in the intensifying AI arms race.
Botscanner
A service of AutoBench
Botscanner is how you use that evaluation idea yourself. Ask one question of several language models at once, then let other models rank the answers so you can compare them on your own case.
Open Botscanner →botscanner.ai is the same service.
Benchmark models. Grade routers.
Use the public leaderboards, read the Router Benchmark, or ask us to run the engine on your models and your traffic. The data is also on Hugging Face.
Frequently Asked Questions
Still have doubts? These fast answers clear up the most common concerns about bringing AutoBench into your workflow.