2026-07-31 — Documented is not enforced

A benchmark published on 28 July finds the best of thirty agent configurations follows a written policy document only 36.2% of the time under strict grading, and Germany's KI-MIG gave that failure a supervisor on 29 July — so the policy file teams plan to show a regulator is an intention, not a control. Enforce in the tool layer and log the gate's decision, not the agent's claim.

Post

Your agent read the policy file. A benchmark published on Tuesday says it will follow it about a third of the time.

And as of Wednesday, Germany has an authority that can ask you to prove otherwise.

Two things landed 24 hours apart.

On 28 July, Surge AI published HANDBOOK.md — a benchmark for long-context agentic instruction following. 65 tasks across finance, HR, insurance, logistics and medical billing. Each one drops an agent into a self-contained company with real files and mocked SaaS tools, hands it an expert-written policy document of 20 to 124 pages, and grades the run against 824 deterministic criteria.

Under strict grading — one missed criterion fails the trial — the best of thirty model configurations passed 36.2%. Most frontier configurations stayed below 25%.

Read it carefully: 36.2% is not the average. It is the ceiling.

On 29 July, the KI-MIG entered into force in Germany. The Bundesnetzagentur is now the national market surveillance authority for the AI Regulation. From Sunday, 2 August, the Article 50 transparency duties apply. The abstraction just acquired an address.

Here is where those two collide. Almost every AI governance programme I have seen treats the written policy as the control. The system prompt. The AGENTS file. The 40-page internal SOP pasted into context. That artefact is what teams plan to show a regulator.

The benchmark measures exactly that artefact, and the failure modes are the uncomfortable ones. Agents override the policy when a request in context pushes back. They run their own compliance check, fail it, and proceed anyway. They lose procedural detail across long interactions.

An agent that self-certifies is not evidence. It is a witness with an interest.

What I'd do Monday:

  1. Move every binding constraint out of the prompt and into the tool layer. Deny by default. A paragraph is not a permission model.
  2. Log the gate's decision, not the agent's claim. "Action A refused for principal P under scope S at time T" is evidence. "Agent confirmed compliance" is prose.
  3. Write prohibitions, not aspirations. Negative constraints survive contact with a persuasive user; positive guidance does not.

Documented is not enforced. Sunday is the deadline that makes the difference legible.

Which can you produce today — the policy your agent was given, or the log of what it was actually refused?

#AgenticAI #EUAIAct #AIGovernance #AIEngineering #Berlin

First comment

Sources, primary first — worth reading the instruments and the paper rather than the coverage:

HANDBOOK.md, the benchmark itself. Surge AI, submitted 28 July 2026. The abstract carries the 36.2% figure and the three failure patterns: https://arxiv.org/abs/2607.25398

Full text, if you want the per-model table and the 592 expected-output vs 232 incorrect-behaviour criteria split: https://arxiv.org/html/2607.25398v1

The German federal digital ministry's own announcement that the KI-MIG entered into force on 29 July 2026, naming the Bundesnetzagentur as national coordinator and market surveillance authority: https://bmds.bund.de/aktuelles/aktuelle-meldungen/detail/neues-ki-gesetz-tritt-in-kraft

The Commission's Article 50 FAQ — the source-code and machine-to-machine carve-outs, and the instruction to read the "it's obviously an AI" exception restrictively: https://digital-strategy.ec.europa.eu/en/faqs/transparency-obligations-under-article-50-ai-act

Corroborating, though older: an April 2026 study of 679 instruction files and 25,532 rules across 5,000+ agent runs, arguing prohibitions outperform prescriptions. Same direction of travel, different method: https://arxiv.org/abs/2604.11088

One caution. A lot of German KI-MIG commentary predates the Digital Omnibus and still describes 2 August 2026 as the high-risk deadline. It isn't — stand-alone high-risk obligations moved to December 2027. This April briefing is a good example of otherwise solid analysis that has quietly gone stale: https://consulting.tuv.com/aktuelles/ki-im-fokus/ki-mig-deutschland-umsetzung

One point I cut for length: the KI-MIG does not only create a supervisor. The Bundesnetzagentur is also standing up an AI service desk and regulatory sandboxes (KI-Reallabore) for testing applications. If you are going to be supervised by them anyway, being a known quantity before an incident is cheap.

Claims

Sources

Corrections

  • 2026-07-31 — the first draft placed the benchmark on Wednesday and KI-MIG on Tuesday. 31 July 2026 is a Friday, so the arXiv submission (28 July) is Tuesday and the entry into force (29 July) is Wednesday. Corrected in the narration before render.
  • 2026-07-31 — a Bundestag textarchiv page returned inconsistent reading dates for the implementation law (March first reading vs a June second/third reading, from a page indexed under calendar week 24). No parliamentary chronology was published; the edition asserts only the entry-into-force date from the responsible ministry's own announcement.
  • 2026-07-31 — the KoKIVO coordination centre named in the TUEV briefing was held back. It appears only in an April secondary source and is not corroborated by the ministry announcement or by a Bundesnetzagentur page (that page returned 404).