CLAIM-2026-10-01-06

AutomationBench-AA, Artificial Analysis's variant, covers 657 tasks across the same six domains using simulated Gmail, Google Sheets, Slack, Salesforce, Zendesk, Jira and HubSpot, and scores the average share of each task's objectives a model completes without triggering guardrail violations rather than full-task completion. Its top entries are Claude Sonnet 5.5 at 71.3%, Claude Opus 5.5 at 69.5% and DeepSeek V4.1 Flash at 68.9%. Gemini 4 Argon does not appear on it.

Why it matters

Two benchmarks share a name and do not share a metric. Quoting one number against the other's leaderboard is the conflation that makes agent capability look settled when it is not.

Sources

artificiala2026automatio

CLAIM-2026-10-01-06

AutomationBench-AA, Artificial Analysis's variant, covers 657 tasks across the same six domains using simulated Gmail, Google Sheets, Slack, Salesforce, Zendesk, Jira and HubSpot, and scores the average share of each task's objectives a model completes without triggering guardrail violations rather than full-task completion. Its top entries are Claude Sonnet 5.5 at 71.3%, Claude Opus 5.5 at 69.5% and DeepSeek V4.1 Flash at 68.9%. Gemini 4 Argon does not appear on it.

Why it matters

Two benchmarks share a name and do not share a metric. Quoting one number against the other's leaderboard is the conflation that makes agent capability look settled when it is not.

Sources

artificiala2026automatio