Privacy
Weight 40%Access Model: Strictly Private
"We evaluate a private 657-task held-out split from AutomationBench dataset version 1.0.6"
Primary source (ArtificialAnalysis) ↗
Verified 2026-11-17
AutomationBench-AA scores 10/10 (Tier S, Fortified) on BenchAudit as of November 2026, reflecting a strictly private held-out test set and isolated offline third-party grading.
"We evaluate a private 657-task held-out split from AutomationBench dataset version 1.0.6"
"It tests whether models can complete realistic SaaS workflows that span multiple simulated business apps, using REST APIs as the tool interface."
"Version 4.3 September 2026 Replaced 饾湉鲁-Banking with AutomationBench-AA (5%) in the Agents category"
"Objectives and guardrails are graded using programmatic checks on the final environment state. AutomationBench-AA does not use a separate LLM judge for grading"
| Rank | Model | Official Score | Percentile |
|---|---|---|---|
| #1 | Gemini 4 Neon xhigh Google | 79.3% | 100.0p |
| #2 | Gemini 4 Carbon high Google | 77.8% | 98.0p |
| #3 | Gemini 4 Argon high Google | 77.5% | 95.9p |
| #4 | GPT-6.2 Astra max OpenAI | 77.2% | 93.9p |
| #5 | Claude Fable 5.2 max Anthropic | 76.4% | 91.8p |
| #6 | Gemini 4 Hydrogen xhigh Google | 76.1% | 89.8p |
| #7 | Kimi K3.1 Pro max Kimi | 73.3% | 87.8p |
| #8 | Qwen4 Max xhigh Alibaba | 72.6% | 85.7p |
| #9 | MiniMax-M3.1-Flash max MiniMax | 72.5% | 83.7p |
| #10 | Claude Sonnet 5.5 max Anthropic | 71.8% | 81.6p |
| #11 | Muse Wonder Full max Meta | 71.3% | 79.6p |
| #12 | GPT-6.2 Sol max OpenAI | 71.3% | 77.6p |
| #13 | Claude Opus 5.5 max Anthropic | 69.5% | 75.5p |
| #14 | DeepSeek V4.1 Flash max DeepSeek | 68.9% | 73.5p |
| #15 | GPT-6 Astra max OpenAI | 68.5% | 71.4p |
Benchmark maintainers can embed this live SVG badge in GitHub READMEs or documentation:
[](https://benchaudit.org/benchmarks/automationbench-aa/)