CLAIM-2026-10-01-03

AutomationBench is a Zapier benchmark (Daniel Shepard and Robin Salimans, both Zapier, April 2026) evaluating AI agents on cross-application workflow orchestration via REST APIs across Sales, Marketing, Operations, Support, Finance and HR; it comprises 600 public tasks plus 600+ private, roughly 500 endpoints across 47 apps, and grading is programmatic and end-state only with binary pass/fail and no partial credit.

Why it matters

End-state-only, no-partial-credit grading means a failed run is not work left undone — it is writes already committed to live systems with the task unfinished. That is the failure mode to design against.

Sources

arxiv2026automatio
github2026zapieraut

CLAIM-2026-10-01-03

AutomationBench is a Zapier benchmark (Daniel Shepard and Robin Salimans, both Zapier, April 2026) evaluating AI agents on cross-application workflow orchestration via REST APIs across Sales, Marketing, Operations, Support, Finance and HR; it comprises 600 public tasks plus 600+ private, roughly 500 endpoints across 47 apps, and grading is programmatic and end-state only with binary pass/fail and no partial credit.

Why it matters

Sources

arxiv2026automatio
github2026zapieraut