Quirin Schlegel

Loading

Nobody else ran it — Gemini 4 Argon claims first place on a benchmark with two metrics, and the duty that would ask you to supervise is fourteen months out

On 30 September 2026 Google announced Gemini 4 Argon, released first to trusted cyber defenders through its Fairwind Program, and reported first place on AutomationBench with a score of 51.3% alongside 77.9% on DeepSWE v1.1 and a tie for first on CWE-bench v1 at 68%. The figure does not stand alone: AutomationBench is two benchmarks sharing a name. Zapier's, whose paper (Shepard and Salimans, Zapier, April 2026) defines grading as programmatic and end-state only with binary pass/fail over 600 public tasks and roughly 500 endpoints across 47 apps, is topped on its own public leaderboard by Claude Opus 4.8 at 30.33%, and reported the best frontier models below 10% at introduction. Artificial Analysis's AutomationBench-AA scores the average share of each task's objectives completed without guardrail violations over 657 tasks, and its leaders sit at 71.3%, 69.5% and 68.9%. A score of 51.3% is therefore consistent with first place only on the strict metric, and Gemini 4 Argon appears on neither public board — so the ranking is today the vendor's own, unreproduced. The practitioner consequence follows from the grading rule rather than the ranking: where a task is scored on end state alone, a failed run has already written into a CRM, an inbox and a chat tool with the job unfinished, which makes replayability, per-system write scoping and end-state reconciliation the controls that matter rather than step logs. The regulatory shape is a gap: the Digital Omnibus, Regulation (EU) 2026/1744, moved Annex III high-risk to 2 December 2027, taking Article 14 human oversight with it, while Article 50 transparency has applied since 2 August 2026 with machine-readable marking from 2 December 2026 — a duty to declare the machine, not to check it. The limb that reaches a wrong record written by an agent is GDPR Article 5(1)(d) accuracy, which carries no deferral.

1 October 20269 verified claims7 sources

A contact created, an email sent, and the deal stage exactly where it was on Friday. That is what a failure looks like when the grading is end-state.

On Wednesday Google shipped Gemini 4 Argon — to trusted cyber defenders first, through its Fairwind Program — and reported first place on AutomationBench at 51.3%.

Flat:

1. The headline numbers are Google's own: 77.9% on DeepSWE v1.1, a tie for first on CWE-bench v1 at 68%, and #1 on AutomationBench at 51.3%.

2. Here is the number that belongs beside it. Zapier's public AutomationBench leaderboard — 600 tasks, strict pass rate, 1.0 only if every assertion passes — tops out at Claude Opus 4.8 on 30.33%. When the benchmark was introduced in April, its paper put the best frontier models below 10%.

3. Before you conclude frontier agents now finish half your business workflows: there are two AutomationBenches. Zapier's grades binary, end-state, no partial credit. Artificial Analysis's AutomationBench-AA grades the average share of each task's objectives completed without triggering guardrail violations, and its top three sit at 71.3, 69.5 and 68.9. A 51.3% can only be first place on the strict metric — and Argon appears on neither public board. The claim may be perfectly good. Today it is Google's word, on Google's run.

4. And if your instinct is that a rule catches the gap: the duty that would ask you to supervise an agent is not live. Annex III high-risk — where the HR workflows in this benchmark would sit — moved to 2 December 2027 under the Digital Omnibus, Regulation (EU) 2026/1744, and Article 14 human oversight went with it. What binds today is Article 50 transparency, in force since 2 August 2026, with the machine-readable marking limb 62 days out on 2 December. That one makes you declare the machine, not check it. The limb that bites on a wrong CRM record is older and has no deferral: GDPR Article 5(1)(d), accuracy.

My Monday: make every agent write replayable, so a rerun is safe rather than doubled. One write scope per agent, per system. Reconcile the end state, not the step log — the step log is what the agent believed happened.

A benchmark score is a claim about a metric, and about who ran it.

Which of your numbers has a scorekeeper who is not the vendor?

#AgenticAI #EUAIAct #GDPR #AIGovernance #DevSecOps

Evidence

What this rests on

Every factual claim in the edition above, stated on its own and checked before publication. If one of these is wrong, the argument is wrong — so they are listed where you can check them.

Google announced Gemini 4 Argon on 30 September 2026, rolling out first to a set of trusted cyber defenders through its Fairwind Program, with broader access to follow for paid API customers and Google AI Ultra subscribers.
Why it mattersA staged release to a named programme is still a supply in the course of a commercial activity. If the preview reaches the Union, the GPAI duties attach at that point, not at general availability.
bloggoogle2026gemini · axios2026googleb
In its own announcement Google reports Gemini 4 Argon at 77.9% on DeepSWE v1.1, ranked #1 on AutomationBench with a score of 51.3%, tied for first on CWE-bench v1 at 68%, and 91.7% on LVBench; introductory API pricing is $2 per million input tokens and $10 per million output tokens, rising to $4 and $20 after the introductory period.
Why it mattersThese are the numbers that will be quoted in slide decks next week. Every one of them is the vendor's own measurement, and knowing that is the whole of the practitioner's edge here.
bloggoogle2026gemini
AutomationBench is a Zapier benchmark (Daniel Shepard and Robin Salimans, both Zapier, April 2026) evaluating AI agents on cross-application workflow orchestration via REST APIs across Sales, Marketing, Operations, Support, Finance and HR; it comprises 600 public tasks plus 600+ private, roughly 500 endpoints across 47 apps, and grading is programmatic and end-state only with binary pass/fail and no partial credit.
Why it mattersEnd-state-only, no-partial-credit grading means a failed run is not work left undone — it is writes already committed to live systems with the task unfinished. That is the failure mode to design against.
arxiv2026automatio · github2026zapieraut
The AutomationBench paper states that even the best frontier models currently score below 10%, reporting Opus 4.7 at 9.9%, Gemini 3.1 Pro at 9.6% and GPT 5.4 at 7.6%.
Why it mattersThe April baseline is the comparator that makes the new figure legible. Without it, 51.3% is a number with nothing to measure it against.
arxiv2026automatio
Zapier's public AutomationBench leaderboard over the 600-task public split, using a strict pass rate scored 1.0 only if every assertion passes, is topped by Claude Opus 4.8 at 30.33%, followed by GPT-5.6 Sol at 29.17%, GPT-5.6 Terra and Claude Fable 5 at 25.83%, Claude Sonnet 5 at 24.00%, GLM 5.2 at 20.33% and Gemini 3.5 Flash at 14.83%. Gemini 4 Argon does not appear on it.
Why it mattersThe best independently listed score on the strict metric is 30.33%. Any planning assumption built on a vendor's 51.3% is 21 points ahead of what a third party has published.
github2026zapieraut
AutomationBench-AA, Artificial Analysis's variant, covers 657 tasks across the same six domains using simulated Gmail, Google Sheets, Slack, Salesforce, Zendesk, Jira and HubSpot, and scores the average share of each task's objectives a model completes without triggering guardrail violations rather than full-task completion. Its top entries are Claude Sonnet 5.5 at 71.3%, Claude Opus 5.5 at 69.5% and DeepSeek V4.1 Flash at 68.9%. Gemini 4 Argon does not appear on it.
Why it mattersTwo benchmarks share a name and do not share a metric. Quoting one number against the other's leaderboard is the conflation that makes agent capability look settled when it is not.
artificiala2026automatio
AI Act Article 52(1) requires a provider to notify the Commission without delay and in any event within two weeks after a general-purpose AI model meets the systemic-risk classification requirement; Article 52(2) and (3) allow the provider to contest the classification and the Commission to reject insufficiently substantiated arguments; Article 52(6) requires the Commission to publish and keep updated a list of general-purpose AI models with systemic risk.
Why it mattersThe notification clock is keyed to the model meeting the threshold, not to launch day. A provider cannot defer the duty by limiting availability, and 52(6) makes the result checkable.
artificiali2026article
AI Act Article 50 transparency obligations have applied since 2 August 2026 and the machine-readable marking obligation for generated content applies from 2 December 2026. The Digital Omnibus, Regulation (EU) 2026/1744, in force 27 July 2026, moved Annex III high-risk obligations to 2 December 2027 and Annex I to 2 August 2028, so Article 14 human oversight for Annex III systems including employment and workers management is not binding until 2 December 2027.
Why it mattersThe duty in force asks you to declare the machine; the duty that would ask you to supervise it is fourteen months out. Everything in that gap is your own change control.
artificiali2026article
GDPR Article 5(1)(d) requires personal data to be accurate and, where necessary, kept up to date, and that every reasonable step be taken to ensure that inaccurate personal data are erased or rectified without delay. It is subject to no deferral under the Digital Omnibus.
Why it mattersThe accuracy principle reaches a wrong record written by an agent today, in every member state, with no deferral to wait for.
gdprinfoeu2026art

Sources

Artificial Analysisprimary source
artificiala2026automatio
Original Archived PDFretrieved 1 October 2026 · published 1 January 2026
artificialintelligenceact.euprimary source
artificiali2026article
Original Archived PDFretrieved 1 October 2026 · published 1 January 2026
arXivprimary source
arxiv2026automatio
Original Archived PDFretrieved 1 October 2026 · published 1 April 2026
Axiossecondary
axios2026googleb
Original Archived PDFretrieved 1 October 2026 · published 30 September 2026
blog.googleprimary source
bloggoogle2026gemini
Original Archived PDFretrieved 1 October 2026 · published 30 September 2026
gdpr-info.euprimary source
gdprinfoeu2026art
Original Archived PDFretrieved 1 October 2026 · published 1 January 2026
GitHubprimary source
github2026zapieraut
Original Archived PDFretrieved 1 October 2026 · published 1 January 2026

Every source for this edition, as one PDF BibTeX — all editions

Corrections

What changed after publication

No corrections. Two namespace hazards caught during the run and handled rather than published: the cite key axios2026google already existed in the vault for a different Axios article of 18 September, so this edition's Axios source is keyed axios2026googleb; and the AutomationBench identifier was disambiguated against both the Zapier paper and the Artificial Analysis variant before any figure was used, because a keyword match on the benchmark name returns two different scoring metrics.