Submit Models
Point to GPT-4o, Claude, Gemini, Llama 3—or your own private endpoint—and specify the subject areas you care about.
Delivering Transparency
in LLM Benchmarking
We use multi-LLM evaluation for accurate and unbiased evaluation of LLM quality, cost and speed. AutoBench resists gaming by changing at each run.
Our system uses 20+ LLM models to generate granular benchmarks that score 90%+ correlation with AAII and 80%+ with LMArena.
Leaderboards
QUALITY
Average Score combining domain-specific Autobench scores; Higher is better
- 4.51
- 4.49
- 4.48
- 4.42
- 4.41
- 4.33
- 4.32
- 4.31
- 4.27
- 4.24
- 4.18
- 4.18
- 4.18
- 4.17
- 4.17
- 4.06
- 3.98
- 3.98
- 3.95
- 3.95
- 3.88
- 3.66
- 3.64
- 3.59
- 3.54
- 3.49
PRICE
USD cent per average answer; Lower is better
- 0.02
- 0.02
- 0.03
- 0.05
- 0.08
- 0.08
- 0.09
- 0.12
- 0.14
- 0.18
- 0.24
- 0.24
- 0.36
- 0.45
- 0.63
- 0.63
- 0.64
- 0.83
- 0.87
- 0.91
- 1.59
- 1.71
- 1.85
- 2.92
- 4.37
- 9.13
LATENCY
Average Latency in Seconds; Lower is better
- 5.29s
- 7.53s
- 7.74s
- 10.65s
- 11.52s
- 17.54s
- 19.16s
- 24.36s
- 26.12s
- 27.01s
- 29.72s
- 32.86s
- 33.67s
- 39.05s
- 40.3s
- 48.62s
- 48.71s
- 60.96s
- 61.12s
- 63.9s
- 65.02s
- 65.03s
- 65.9s
- 66.5s
- 68.34s
- 72.64s
- 80.74s
- 90s
- 119.17s
Benchmarks
Model | Average (All Topics) | Coding | Creative Writing | Current News | General Culture | Grammar | History | Logics | Math | Science | Technology |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 90.00s (#32) | 120.97s (#30) | 50.04s (#33) | 78.84s (#33) | 55.10s (#33) | 56.65s (#33) | 73.40s (#32) | 156.60s (#24) | 151.20s (#30) | 83.20s (#33) | 76.55s (#33) | |
| 65.02s (#24) | 69.47s (#18) | 29.26s (#28) | 65.40s (#32) | 45.96s (#31) | 46.64s (#29) | 75.24s (#33) | 114.36s (#18) | 58.67s (#20) | 72.55s (#32) | 57.17s (#30) | |
| 119.17s (#33) | 238.20s (#33) | 38.15s (#31) | 62.97s (#30) | 48.22s (#32) | 51.71s (#31) | 61.99s (#31) | 302.30s (#33) | 271.52s (#33) | 62.54s (#31) | 66.13s (#31) | |
| 78.79s (#30) | 180.14s (#32) | 33.74s (#30) | 65.22s (#31) | 45.30s (#30) | 54.66s (#32) | 53.71s (#28) | 122.66s (#20) | 138.29s (#28) | 60.43s (#30) | 72.91s (#32) | |
| 65.90s (#26) | 102.52s (#27) | 25.72s (#22) | 48.42s (#26) | 31.52s (#23) | 35.42s (#23) | 44.41s (#25) | 159.72s (#25) | 84.27s (#21) | 52.17s (#29) | 56.72s (#29) | |
| 65.03s (#25) | 83.39s (#21) | 29.26s (#27) | 49.16s (#27) | 36.70s (#28) | 36.99s (#24) | 47.21s (#26) | 166.92s (#26) | 93.46s (#23) | 49.47s (#28) | 52.59s (#27) | |
| 60.96s (#20) | 92.70s (#23) | 28.46s (#25) | 49.77s (#28) | 33.72s (#26) | 41.45s (#27) | 48.44s (#27) | 138.43s (#23) | 124.39s (#25) | 48.72s (#27) | 50.10s (#25) | |
| 66.50s (#27) | 98.42s (#26) | 38.22s (#32) | 52.23s (#29) | 38.22s (#29) | 48.28s (#30) | 54.48s (#30) | 136.66s (#22) | 86.48s (#22) | 48.33s (#26) | 53.91s (#28) | |
| 48.62s (#18) | 97.99s (#25) | 31.75s (#29) | 44.91s (#24) | 36.35s (#27) | 30.17s (#20) | 53.86s (#29) | 38.46s (#11) | 32.15s (#9) | 48.13s (#25) | 52.28s (#26) | |
| 80.74s (#31) | 147.04s (#31) | 20.92s (#21) | 43.49s (#23) | 31.81s (#24) | 38.41s (#25) | 42.51s (#23) | 224.90s (#32) | 165.52s (#32) | 43.16s (#24) | 49.76s (#24) | |
| 63.90s (#23) | 70.42s (#19) | 25.94s (#23) | 46.46s (#25) | 29.61s (#22) | 26.53s (#19) | 42.66s (#24) | 194.81s (#28) | 112.94s (#24) | 41.25s (#23) | 46.78s (#23) | |
| 61.12s (#21) | 67.73s (#17) | 19.92s (#20) | 31.32s (#18) | 32.18s (#25) | 32.24s (#22) | 31.20s (#19) | 197.42s (#29) | 132.55s (#26) | 40.17s (#22) | 31.52s (#16) | |
| 61.54s (#22) | 88.51s (#22) | 28.89s (#26) | 29.22s (#16) | 22.75s (#17) | 32.21s (#21) | 36.15s (#21) | 174.68s (#27) | 134.89s (#27) | 38.29s (#21) | 32.33s (#17) | |
| 72.64s (#29) | 119.99s (#29) | 27.91s (#24) | 34.78s (#21) | 25.85s (#21) | 38.81s (#26) | 37.06s (#22) | 204.23s (#30) | 157.67s (#31) | 38.20s (#20) | 41.63s (#20) | |
| 68.34s (#28) | 105.91s (#28) | 13.72s (#13) | 31.22s (#17) | 19.54s (#15) | 45.69s (#28) | 30.92s (#17) | 206.30s (#31) | 140.95s (#29) | 38.18s (#19) | 44.16s (#22) | |
| 33.67s (#15) | 75.45s (#20) | 18.12s (#19) | 34.98s (#22) | 23.57s (#20) | 18.97s (#12) | 35.19s (#20) | 19.96s (#7) | 24.19s (#7) | 36.59s (#18) | 34.59s (#18) | |
| 29.72s (#12) | 62.27s (#16) | 12.47s (#11) | 27.51s (#13) | 18.32s (#13) | 19.29s (#13) | 25.02s (#13) | 24.37s (#9) | 40.47s (#14) | 35.35s (#17) | 23.28s (#10) | |
| 32.64s (#13) | 46.36s (#10) | 15.18s (#15) | 28.78s (#14) | 22.91s (#18) | 21.54s (#15) | 27.26s (#15) | 67.43s (#13) | 33.04s (#11) | 31.87s (#16) | 25.02s (#13) | |
| 48.71s (#19) | 95.88s (#24) | 16.73s (#17) | 32.10s (#20) | 18.35s (#14) | 24.89s (#18) | 29.12s (#16) | 133.86s (#21) | 52.62s (#19) | 31.52s (#15) | 36.19s (#19) | |
| 40.30s (#17) | 55.10s (#13) | 17.82s (#18) | 32.07s (#19) | 23.04s (#19) | 24.28s (#17) | 31.00s (#18) | 88.90s (#17) | 51.26s (#18) | 29.38s (#14) | 43.42s (#21) | |
| 39.05s (#16) | 56.98s (#14) | 16.40s (#16) | 26.73s (#12) | 19.81s (#16) | 21.71s (#16) | 23.26s (#12) | 116.34s (#19) | 41.53s (#16) | 28.65s (#13) | 26.42s (#14) | |
| 27.01s (#11) | 49.98s (#11) | 11.36s (#9) | 28.99s (#15) | 18.29s (#12) | 14.40s (#10) | 22.62s (#10) | 25.45s (#10) | 36.60s (#12) | 27.85s (#12) | 28.81s (#15) | |
| 26.12s (#10) | 45.77s (#9) | 12.96s (#12) | 17.72s (#9) | 13.61s (#9) | 20.44s (#14) | 20.45s (#9) | 40.52s (#12) | 32.54s (#10) | 25.81s (#11) | 23.90s (#12) | |
| 24.36s (#9) | 51.77s (#12) | 14.28s (#14) | 23.60s (#10) | 17.35s (#11) | 13.37s (#9) | 25.84s (#14) | 18.14s (#6) | 24.62s (#8) | 25.15s (#10) | 21.90s (#9) | |
| 32.86s (#14) | 58.02s (#15) | 11.49s (#10) | 24.72s (#11) | 14.47s (#10) | 15.96s (#11) | 23.16s (#11) | 80.41s (#15) | 46.81s (#17) | 21.99s (#9) | 23.40s (#11) | |
| 10.87s (#5) | 20.22s (#6) | 6.43s (#5) | 9.19s (#7) | 7.53s (#7) | 7.99s (#7) | 9.81s (#7) | 11.12s (#5) | 13.07s (#6) | 12.25s (#8) | 8.23s (#6) | |
| 11.52s (#6) | 17.21s (#5) | 9.50s (#8) | 13.16s (#8) | 11.23s (#8) | 9.27s (#8) | 13.71s (#8) | 7.91s (#4) | 9.50s (#3) | 10.84s (#7) | 11.28s (#8) | |
| 10.65s (#4) | 21.57s (#7) | 4.76s (#2) | 7.82s (#6) | 5.84s (#5) | 6.17s (#5) | 8.32s (#5) | 21.06s (#8) | 11.33s (#4) | 8.72s (#6) | 6.83s (#3) | |
| 17.54s (#7) | 11.46s (#3) | 7.20s (#6) | 7.52s (#4) | 5.75s (#4) | 6.11s (#4) | 8.70s (#6) | 79.27s (#14) | 37.96s (#13) | 7.09s (#5) | 7.28s (#4) | |
| 7.74s (#3) | 10.84s (#2) | 5.95s (#4) | 6.78s (#3) | 6.31s (#6) | 6.00s (#3) | 7.15s (#3) | 7.76s (#3) | 12.17s (#5) | 7.04s (#4) | 7.41s (#5) | |
| 7.53s (#2) | 12.48s (#4) | 7.58s (#7) | 7.52s (#5) | 5.67s (#3) | 6.53s (#6) | 7.34s (#4) | 5.86s (#2) | 7.27s (#2) | 6.68s (#3) | 6.78s (#2) | |
| 19.16s (#8) | 26.18s (#8) | 2.86s (#1) | 5.62s (#1) | 3.62s (#1) | 3.98s (#1) | 6.84s (#2) | 86.53s (#16) | 41.51s (#15) | 6.41s (#2) | 9.55s (#7) | |
| 5.29s (#1) | 7.10s (#1) | 4.79s (#3) | 5.85s (#2) | 4.71s (#2) | 4.34s (#2) | 5.51s (#1) | 4.88s (#1) | 4.86s (#1) | 4.91s (#1) | 5.33s (#1) |
HOW IT WORKS
AutoBench operates through a fully automated, iterative process designed for robustness and statistical significance.
01 02 Run Benchmarks
The engine writes difficulty-balanced prompts, solicits answers from each model, and quality-checks every response automatically.
03 Collect Metrics
Every model anonymously judges its peers; a weighting algorithm refines scores until the leaderboard stabilises.
04 Analyze Results
Download a ready-to-share CSV plus an interactive dashboard that plugs into Hugging Face Spaces or your internal BI tools.
Validation: Proven Accuracy at Scale
AutoBench’s effectiveness is not theoretical. The results from its public runs demonstrate both unprecedented scale and exceptionally high correlation with industry-standard benchmarks
- 92.17%AAII
- 86.85%LMArena
- 75.44%MMLU-Plus
- 33ranked models
- 13ranking models
- 400kranks
- 400iterations
- 13kanswers
- ≈250Mgenerated tokens
- ≈5Bevaluated tokens
AutoBench your models today
We invite you to explore the code, run the benchmark, contribute to its development, and join the discussion on the future of LLM evaluation. Explore our resources on Hugging Face.
Contact usAutoBench for Enterprises and LLM Labs
For Enterprises with Large-Scale LLM Consumption
Benchmark your Use Cases
Large corporations project billions in LLM API calls, but relying on a single model for all tasks leads to massive inefficiencies. AutoBench evaluates models on your internal use cases and data, identifying the optimal model for tasks like sentiment analysis, document summarization, or customer support.
See $ Trade-Offs Instantly
Gain immediate visibility into cost-quality trade-offs. By analyzing performance metrics like average answer cost and P99 duration, AutoBench reveals how switching models can save an estimated 20%+ on LLM expenditure without sacrificing quality.
Switch & Monitor
Seamlessly switch to cost-effective models and monitor ongoing performance. Our enterprise-specific benchmarks ensure continuous optimization, preventing overpayments and improving reliability in high-volume AI deployments.
Empowering LLM Lab R&D
Benchmark your Use Cases
With over 20 major labs competing and a $50M TAM for R&D enablement in 2025, granular evaluation is critical. AutoBench offers private, domain-focused benchmarks to reveal weaknesses in areas like advanced reasoning or specific coding.
See $ Trade-Offs Instantly
Get instant, nuanced views of performance trade-offs through collective LLM judging. Backed by ~300,000 ranks and high correlations (e.g., 86.85% with human preference), it provides actionable data to refine models efficiently.
Switch & Monitor
Monitor progress and switch training strategies with ease. Our scalable framework supports continuous custom runs, helping labs adapt architectures and data for better outcomes in the intensifying AI arms race.
Frequently Asked Questions
Still have doubts? These fast answers clear up the most common concerns about bringing AutoBench into your workflow.