AutoBench Router Benchmark 2 - November 2026
November re-audit on the same frozen answer matrix. Adds OpenRouter's rebuilt Auto Router, Conifer, and dari (select-only). Accuracy is the share of questions where the chosen model is statistically as good as the best available answer.
Router BenchmarkQuality versus cost
Dashed lines are Always-Sol, Always-Luna, and Random. Green points are AutoBench. Violet is OpenRouter's November Auto Router. Blue is Conifer. Orange is dari. Grey points are October when that board includes them. Amber points collapsed onto one model. Whiskers on November points are 95% bootstrap intervals.
Accuracy is the share of questions where the chosen model is statistically as good as the best available answer.
Board
| Pass | Vendor | ARI | n | Accuracy | 95% CI | USD / question | Top-model share | Collapsed |
|---|---|---|---|---|---|---|---|---|
| Cheapest-adequate | Baseline | 88.0 | 252 | 100.0% | 100.0–100.0 | $0.0018 | — | |
| Oracle | Baseline | 71.4 | 252 | 100.0% | 100.0–100.0 | $0.0341 | — | |
| Quality | AutoBench | 60.8 | 245 | 88.6% | 83.7–91.8 | $0.0097 | 66.3% | |
| Balanced | AutoBench | 60.3 | 245 | 87.8% | 83.3–91.8 | $0.0066 | 59.9% | |
| Cost | AutoBench | 59.8 | 242 | 86.8% | 83.1–91.3 | $0.0037 | 78.9% | |
| Always-Luna | Baseline | 56.1 | 251 | 85.3% | 80.9–89.6 | $0.0043 | — | |
| Always-Sol | Baseline | 48.9 | 248 | 87.5% | 83.5–91.5 | $0.0787 | — | |
| ND k0 | Not Diamond | 48.9 | 248 | 87.5% | 83.5–91.5 | $0.0787 | 100.0% | Yes |
| Azure balanced | Azure Model Router | 47.0 | 179 | 82.7% | 78.8–89.4 | $0.0204 | 48.0% | |
| Azure quality | Azure Model Router | 47.0 | 238 | 85.3% | 80.3–89.1 | $0.0647 | 40.9% | |
| dari | dari | 39.3 | 244 | 79.1% | 73.8–84.0 | $0.0220 | 14.7% | |
| Auto high (Aug) | OpenRouter | 39.2 | 244 | 79.1% | 73.4–83.6 | $0.0230 | 57.9% | |
| ND k4 | Not Diamond | 39.0 | 249 | 80.3% | 75.5–85.2 | $0.0600 | 77.0% | |
| Auto max (Aug) | OpenRouter | 37.1 | 240 | 78.3% | 72.9–83.3 | $0.0289 | 78.2% | |
| Conifer | Conifer | 30.5 | 250 | 74.8% | 69.2–80.0 | $0.0052 | 76.6% | |
| Auto medium (Aug) | OpenRouter | 26.7 | 246 | 73.6% | 67.1–78.0 | $0.0131 | 78.2% | |
| ND native | Not Diamond | 21.3 | 250 | 72.0% | 66.0–77.2 | $0.0597 | 63.9% | |
| NadirClaw | NadirClaw | 16.9 | 249 | 70.3% | 64.3–75.9 | $0.0514 | 58.7% | |
| Orca quality | OrcaRouter | 3.5 | 245 | 65.7% | 59.2–71.0 | $0.0005 | 100.0% | Yes |
| Auto low (Aug) | OpenRouter | 3.5 | 245 | 65.7% | 59.2–71.0 | $0.0006 | 98.4% | Yes |
| Auto Router high | OpenRouter | 1.6 | 244 | 65.2% | 59.0–70.9 | $0.0053 | 25.0% | |
| Always-cheapest | Baseline | 0.0 | 252 | 44.4% | 38.5–50.4 | $0.0002 | — | |
| Orca balanced | OrcaRouter | 0.0 | 252 | 39.3% | 33.3–45.2 | $0.0002 | 99.6% | Yes |
| Orca cheapest | OrcaRouter | 0.0 | 252 | 39.3% | 33.3–45.2 | $0.0002 | 99.2% | Yes |
| ND k5 | Not Diamond | 0.0 | 251 | 46.6% | 40.2–52.6 | $0.0009 | 82.9% | |
| Auto Router medium | OpenRouter | 0.0 | 247 | 57.9% | 51.8–64.0 | $0.0021 | 43.3% | |
| Auto Router default | OpenRouter | 0.0 | 246 | 58.1% | 52.0–64.2 | $0.0024 | 39.3% | |
| Auto Router low | OpenRouter | 0.0 | 247 | 60.3% | 54.3–66.4 | $0.0025 | 38.5% | |
| Brick | Brick | 0.0 | 241 | 63.9% | 57.7–69.7 | $0.0031 | 80.2% | |
| Azure cost | Azure Model Router | 0.0 | 74 | 45.9% | 33.8–56.8 | $0.0091 | 59.1% | |
| Cascade | Cascade | 0.0 | 235 | 60.9% | 53.2–66.0 | $0.0104 | 58.3% | |
| Random | Baseline | 0.0 | 252 | 64.7% | 58.7–70.6 | $0.0115 | 7.1% |
ARI is the AutoBench Router Index at β = 0.2. The board is ordered by it, highest first. It runs from 0 to 100 and is higher when a pass is both accurate and cheap. A pass no better than random scores 0. How ARI is calculated. Accuracy is the share of questions where the chosen model is statistically as good as the best available answer.
Same-pick
Median cross-vendor same-pick is 0 across 59 pairs. The grid is the 12-pass comparison from the report. A cell is the share of questions on which the two passes named the same model.
| Orca cheap | Orca bal | Orca qual | Azure cost | Azure bal | Azure qual | NadirClaw | Brick | Cascade | ND k5 | Auto high | Auto low | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Orca cheap | 1.00 | 0.99 | 0.01 | 0.14 | 0.01 | 0.00 | 0.02 | 0.00 | 0.01 | 0.00 | 0.00 | 0.01 |
| Orca bal | 0.99 | 1.00 | 0.00 | 0.14 | 0.01 | 0.00 | 0.02 | 0.00 | 0.00 | 0.00 | 0.00 | 0.01 |
| Orca qual | 0.01 | 0.00 | 1.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.80 | 0.58 | 0.00 | 0.00 | 0.98 |
| Azure cost | 0.14 | 0.14 | 0.00 | 1.00 | 0.19 | 0.03 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| Azure bal | 0.01 | 0.01 | 0.00 | 0.19 | 1.00 | 0.11 | 0.01 | 0.00 | 0.00 | 0.00 | 0.00 | 0.01 |
| Azure qual | 0.00 | 0.00 | 0.00 | 0.03 | 0.11 | 1.00 | 0.12 | 0.00 | 0.00 | 0.00 | 0.01 | 0.00 |
| NadirClaw | 0.02 | 0.02 | 0.00 | 0.00 | 0.01 | 0.12 | 1.00 | 0.00 | 0.00 | 0.34 | 0.00 | 0.00 |
| Brick | 0.00 | 0.00 | 0.80 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.48 | 0.00 | 0.00 | 0.79 |
| Cascade | 0.01 | 0.00 | 0.58 | 0.00 | 0.00 | 0.00 | 0.00 | 0.48 | 1.00 | 0.00 | 0.00 | 0.57 |
| ND k5 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.34 | 0.00 | 0.00 | 1.00 | 0.00 | 0.00 |
| Auto high | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.01 | 0.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.00 |
| Auto low | 0.01 | 0.01 | 0.98 | 0.00 | 0.01 | 0.00 | 0.00 | 0.79 | 0.57 | 0.00 | 0.00 | 1.00 |
Scored on the answer matrix published as AutoBench Run 6.