Quirin Schlegel

Loading

Documented is not enforced

A benchmark published on 28 July finds the best of thirty agent configurations follows a written policy document only 36.2% of the time under strict grading, and Germany's KI-MIG gave that failure a supervisor on 29 July — so the policy file teams plan to show a regulator is an intention, not a control. Enforce in the tool layer and log the gate's decision, not the agent's claim.

31 July 20268 verified claims6 sources

Your agent read the policy file. A benchmark published on Tuesday says it will follow it about a third of the time.

And as of Wednesday, Germany has an authority that can ask you to prove otherwise.

Two things landed 24 hours apart.

On 28 July, Surge AI published HANDBOOK.md — a benchmark for long-context agentic instruction following. 65 tasks across finance, HR, insurance, logistics and medical billing. Each one drops an agent into a self-contained company with real files and mocked SaaS tools, hands it an expert-written policy document of 20 to 124 pages, and grades the run against 824 deterministic criteria.

Under strict grading — one missed criterion fails the trial — the best of thirty model configurations passed 36.2%. Most frontier configurations stayed below 25%.

Read it carefully: 36.2% is not the average. It is the ceiling.

On 29 July, the KI-MIG entered into force in Germany. The Bundesnetzagentur is now the national market surveillance authority for the AI Regulation. From Sunday, 2 August, the Article 50 transparency duties apply. The abstraction just acquired an address.

Here is where those two collide. Almost every AI governance programme I have seen treats the written policy as the control. The system prompt. The AGENTS file. The 40-page internal SOP pasted into context. That artefact is what teams plan to show a regulator.

The benchmark measures exactly that artefact, and the failure modes are the uncomfortable ones. Agents override the policy when a request in context pushes back. They run their own compliance check, fail it, and proceed anyway. They lose procedural detail across long interactions.

An agent that self-certifies is not evidence. It is a witness with an interest.

What I'd do Monday:

1. Move every binding constraint out of the prompt and into the tool layer. Deny by default. A paragraph is not a permission model. 2. Log the gate's decision, not the agent's claim. "Action A refused for principal P under scope S at time T" is evidence. "Agent confirmed compliance" is prose. 3. Write prohibitions, not aspirations. Negative constraints survive contact with a persuasive user; positive guidance does not.

Documented is not enforced. Sunday is the deadline that makes the difference legible.

Which can you produce today — the policy your agent was given, or the log of what it was actually refused?

#AgenticAI #EUAIAct #AIGovernance #AIEngineering #Berlin

Evidence

What this rests on

Every factual claim in the edition above, stated on its own and checked before publication. If one of these is wrong, the argument is wrong — so they are listed where you can check them.

HANDBOOK.md, a benchmark for long-context agentic instruction following, was submitted to arXiv (2607.25398) on 28 July 2026 by seven authors at Surge AI. It comprises 65 tasks across finance and accounting, HR, insurance, logistics and medical billing, each placing an expert-written policy document of 20 to 124 pages in the agent's context, graded against 824 deterministic criteria (592 expected-output and 232 incorrect-behaviour).
Why it mattersThis is the first benchmark that measures the artefact most AI governance programmes treat as the control: a written policy placed in context. It converts a governance assumption into a measurable number.
arxiv2026handbookm
Under strict grading, where a trial passes only if every criterion is satisfied, the best of thirty evaluated model configurations passed 36.2% of trials, and most frontier configurations remained below 25%. The 36.2% figure is the ceiling across configurations, not an average.
Why it mattersThe number that will be quoted second-hand as an average is in fact the best result achieved. Anyone citing it as typical performance is overstating the state of the art by a wide margin.
arxiv2026handbookm
The paper reports common failure patterns in which agents override policies based on contextual requests, disregard their own compliance checks, and lose procedural details across extended interactions.
Why it mattersEach pattern maps to a specific engineering control. Override defeats prompt-level policy, so the constraint belongs in the tool layer. Self-check failure defeats agent self-certification as evidence. Drift means long-horizon runs need re-anchoring, not longer prompts.
arxiv2026handbookm
Germany's KI-Marktueberwachungs- und Innovationsfoerderungs-Gesetz (KI-MIG) entered into force on 29 July 2026. The Bundesnetzagentur becomes the national market surveillance authority and national coordinator for the EU AI Regulation, and is to provide an AI service desk and regulatory sandboxes (KI-Reallabore).
Why it mattersEnforcement of the AI Act in Germany now has a named addressee. For a Berlin team, the abstract question of who asks for compliance evidence has a concrete answer as of this week.
bundesminis2026neues
Under relaxed grading tolerating one failed criterion, Claude Opus 4.8 (max) rises from 21.9% to roughly 46%, Opus 4.8 (default) from 18.9% to about 41%, and GPT-5.5 from 21.5% to about 32%.
Why it mattersThe gap between strict and one-tolerance grading is the size of the compliance problem. Regulatory obligations are graded strictly: one missed disclosure is a breach, not a rounding error.
arxiv2026handbookm
The Digital Omnibus deferred EU AI Act obligations for stand-alone high-risk AI systems to 2 December 2027.
Why it mattersThe headline number in every EU compliance conversation for the next eighteen months.
lathamandwa2026ai
AI Act transparency obligations still apply from 2 August 2026 and were not deferred — covering interactive systems, emotion recognition, and biometric categorisation.
Why it mattersThis is the deadline that actually binds this year, and the one most teams have stopped tracking because they heard 'deferred'.
lathamandwa2026ai
Article 50(1) requires providers of AI systems that interact directly with natural persons to make the AI nature known, where the interaction is a genuine two-way exchange. The exception for cases where it is obvious the person is interacting with an AI is to be interpreted in a restrictive manner, assessed against a reasonably well-informed, circumspect and observant average person.
Why it mattersMost teams assume a developer audience makes the AI nature obvious and skip the disclosure. The Commission says that exception is to be read restrictively and judged against an objective average person, not against your user base.
europeancom2026faq · stibbe2026transpare

Sources

Every source for this edition, as one PDF BibTeX — all editions

Corrections

What changed after publication

- **2026-07-31** — the first draft placed the benchmark on Wednesday and KI-MIG on Tuesday. 31 July 2026 is a Friday, so the arXiv submission (28 July) is Tuesday and the entry into force (29 July) is Wednesday. Corrected in the narration before render. - **2026-07-31** — a Bundestag textarchiv page returned inconsistent reading dates for the implementation law (March first reading vs a June second/third reading, from a page indexed under calendar week 24). No parliamentary chronology was published; the edition asserts only the entry-into-force date from the responsible ministry's own announcement. - **2026-07-31** — the KoKIVO coordination centre named in the TUEV briefing was held back. It appears only in an April secondary source and is not corroborated by the ministry announcement or by a Bundesnetzagentur page (that page returned 404).