The skill file was the spec — agents satisfy every requirement of a procedure 12.6% to 40.9% of the time, and the live duty that asks about procedure is not in the AI Act

Agent skills are specifications, and a benchmark that scores every requirement in one finds the agents satisfy all of them 12.6% to 40.9% of the time while passing 70.0% to 85.7% of the same rubrics scored one at a time; process rubrics, the record of how the work was done, are the weakest artefact of all, under 3% for the smallest agent, and skills widen rather than close the gap between tiers. The regulatory reflex is AI Act Article 14 human oversight, which moved to 2 December 2027 with Annex III under Regulation (EU) 2026/1744; Article 50 has applied since 2 August 2026 and marks the machine rather than inspecting it. The live limb phrased as a process duty is GDPR Article 32(1)(d) read with Article 5(2), and it reaches only processing of personal data — outside that, the trace is a control you build or do not have.

Post

There is a markdown file in your repo that tells the agent how the job is done. A benchmark just checked whether eight agents actually follow one.

SkillForgeBench (arXiv 2610.14918, v1, 10 October 2026) builds 842 sandboxed tasks across five professional domains, uses agent skills as the execution blueprint, and scores each task against fine-grained completion and process rubrics. In the authors' own numbers:

  1. Strict completion — every requirement satisfied — runs from 12.6% to 40.9% across eight agents, led by Opus 4.8 at 40.9%.
  2. Score the rubrics individually and the same agents pass 70.0% to 85.7%. Requiring all of them at once is what collapses the figure.
  3. Process rubrics are satisfied far less often than completion rubrics — under 3% for Haiku 4.5. The record of how the work was done is the weakest artefact in the run.
  4. Skills help unevenly: +19.2 points for Opus 4.8 against +5.8 for Haiku 4.5. External guidance rewarded the agent that needed it least.

Point 3 is the one with a compliance shape, and the reflex is the wrong article. AI Act Article 14 human oversight is what you reach for, and the Digital Omnibus — Regulation (EU) 2026/1744, in force 27 July 2026 — moved Annex III to 2 December 2027. Article 50 transparency has applied since 2 August 2026, with machine-readable marking from 2 December 2026, but that is a duty to declare the machine, not to inspect its procedure.

The limb already phrased as a process duty is not in the AI Act at all. GDPR Article 32(1)(d) requires a process for regularly testing, assessing and evaluating the effectiveness of your measures, and Article 5(2) requires you to be able to demonstrate it. Where your agent touches personal data, a completion-only gate leaves you nothing to demonstrate with. Where it does not, no duty in force reaches the procedure — so the trace is a control you build or simply do not have.

A skill file is a claim about how the work will go. It is not evidence that it did.

What does your acceptance gate record — the result, or the path?

#AgenticAI #EUAIAct #AIGovernance #SoftwareEngineering #DevTools

First comment

Sources, in order of what they carry.

The paper — 842 tasks, the strict/per-rubric split, the process-rubric collapse and the uneven skill gains: https://arxiv.org/abs/2610.14918

The metadata record I verified the identifier and v1 date against, because a keyword search returns a paper on the subject and not necessarily the one: https://export.arxiv.org/api/query?id_list=2610.14918

AI Act Article 14, human oversight — the article the reflex reaches for, deferred with Annex III to 2 December 2027: https://artificialintelligenceact.eu/article/14/

AI Act Article 50, transparency — applying since 2 August 2026, machine-readable marking from 2 December 2026: https://artificialintelligenceact.eu/article/50/

Regulation (EU) 2026/1744, the Digital Omnibus on AI, in force 27 July 2026 — the instrument that moved the dates: https://eur-lex.europa.eu/eli/reg/2026/1744/oj

GDPR, for Article 32(1)(d) and Article 5(2) — the process duty and the demonstrate duty: https://eur-lex.europa.eu/eli/reg/2016/679/oj

Cut for length: the paper lists no author affiliations on its arXiv record and its HTML full text was rate-limited during this run, so the work is credited to the researchers rather than to an institution — and every figure above is quoted from the abstract's own wording rather than from a table I could not open.

Claims

Sources

AI disclosure

The video for this edition contains AI-generated media: the narration is a synthetic voice, the cover art and any background imagery are generated, and the brand ident animation is generated. The script, the claims and the source verification are human-authored and human-checked. Assets carry IPTC DigitalSourceType and C2PA Content Credentials.