Quirin Schlegel

Loading

The reviewer was big enough — a weak model can audit a strong coding agent when the evidence is checkable, and the Article 14 that binds today is not the one you reach for

A weaker reviewer can audit a stronger coding agent, but only when the evidence it is handed has been independently checked: across a ladder from 8B to 235B parameters, reviewer size is not a consistent predictor of quality, and with official execution evidence two of six reviewers classified all 122 held-out traces correctly. Without official tests - the deployment case - the paper's frozen cascade separates catch from over-rejection by only ten to thirteen points, which is not a gate. The regulatory reflex is the wrong Article 14: AI Act Article 14 human oversight moved to 2 December 2027 with Annex III under Regulation (EU) 2026/1744, while Cyber Resilience Act Article 14 has bound manufacturers since 11 September 2026 and keys its final-report clock to the availability of a corrective measure rather than to the incident.

6 October 202611 verified claims7 sources

A diff in the right file. A summary that sounds certain. A test suite that never asked the question the ticket was about.

That patch gets approved, and the approval means nothing.

On 1 October, researchers at UC Berkeley and Virginia Tech asked when a weaker model can reliably decide whether a coding agent's patch solves its issue. 411 execution-labelled traces from three agents, 101 controlled cases, six reviewer models from Llama-3.1-8B to Qwen3-235B.

Flat:

1. Hand the reviewer official execution evidence and five of six improve on both measures; two classified all 122 held-out traces correctly. Hand them structured-but-unchecked evidence — selected, confident, nothing verified — and both numbers move the wrong way together: more defects caught, more good patches rejected.

2. Before you conclude the fix is a bigger reviewer: reviewer size is not a consistent predictor of quality. The ladder spans 8B to 235B and the ranking is not monotonic. The lever is groundability — whether an independent check signed the evidence — not scale.

3. Official tests don't exist in deployment, so the authors froze a cascade: static errors plus generated tests that first fail on the unpatched repo. On held-out sets, catch 0.76 against over-rejection 0.66 on GPT-5.4, catch 0.80 against 0.67 on Gemini. Ten to thirteen points of separation. That is not a gate.

4. Now the European part, and it is not the article you expect. The oversight duty everyone reaches for is AI Act Article 14 — understand the system's limits well enough to spot anomalies. It moved to 2 December 2027 with Annex III, under Regulation (EU) 2026/1744, the Digital Omnibus. The Cyber Resilience Act has an Article 14 too, and it has bound manufacturers since 11 September 2026: 24 hours to warn on an actively exploited vulnerability, 72 to notify, a final report 14 days after a corrective measure is available. That clock starts at your fix, not your incident.

My Monday: make the gate produce a test that fails on the unpatched repo and passes after. Freeze one evidence format per gate. Track over-rejection next to catch — a gate nobody trusts gets switched off. And record which check signed the approval.

Oversight is not someone senior reading more carefully. It is a question your tooling can answer.

Which of your agent's approvals rests on a check that could have said no?

#AgenticAI #EUAIAct #CyberResilienceAct #CodeReview #AIGovernance

Evidence

What this rests on

Every factual claim in the edition above, stated on its own and checked before publication. If one of these is wrong, the argument is wrong — so they are listed where you can check them.

Why it mattersFixes the identifier and the event date. The submission timestamp is arXiv's own record; the cs.SE listing announced it on 2 October, which is a coverage date and not the event. Affiliations are taken from the work's own front matter, collectively, because the per-author mapping was not readable.
arxiv2026groundabi · arxiv2026html
Why it mattersSets the size of the evidence base and, because the reviewer ladder spans 8B to 235B parameters, makes the scale question answerable at all.
arxiv2026groundabi · arxiv2026html
Why it mattersThis is the upper bound: when the evidence is decisive, review quality is no longer a function of the reviewer. It tells a practitioner the budget belongs in test generation, not in a larger review model.
arxiv2026groundabi · arxiv2026html
Why it mattersBoth rates moving together is the signature of a reviewer becoming more suspicious rather than better informed. A gate that catches more defects and rejects more good work has not improved.
arxiv2026groundabi
Why it mattersDirectly contradicts the default procurement reflex of putting the largest available model on review duty.
arxiv2026groundabi · arxiv2026html
Why it mattersThis is the deployment number, because official tests do not exist when shipping. Catch and over-rejection sit only ten to thirteen points apart, which is the measure of how weakly such a gate discriminates.
arxiv2026groundabi · arxiv2026html
Why it mattersWithout the definitions the figures are unreadable, and over-rejection in particular is easy to misreport as a share of all rejections rather than of acceptable patches.
arxiv2026html
Why it mattersLocates part of the over-rejection upstream of review: if the agent decides when it is done, the gate inherits cases it was never going to judge well.
arxiv2026groundabi
Why it mattersThis is the duty practitioners assume already governs an agent operating on their codebase. It does not yet, and the deferral is fourteen months from this edition.
euaiactexpl2026human · eurlex2026regulatio
Why it mattersThis is the live duty, and its final-report clock is keyed to the availability of a corrective measure rather than to the incident — so an agent-authored patch that a weak review waved through starts a statutory clock on a fix that may not fix anything.
europeancom2026cra · eurlex2024regulatio · cyberresili2026cyber
Why it mattersThe paper is an empirical reading of this sentence: understanding the limits of an agent's patch turned out to depend on whether a decisive check was available, not on the seniority or size of the reviewer.
euaiactexpl2026human

Sources

European Commissionprimary source
CRA reporting obligations
Original Archived PDFretrieved 6 October 2026 · published 11 September 2026
CyberResilienceAct.eusecondary
Cyber Resilience Act explained — scope, classes and deadlines
Original Archived PDFretrieved 6 October 2026 · published 11 September 2026
arXivprimary source
Groundability, Not Scale Alone- When Weak Reviewers Can Audit Strong Coding Agents
Original Archived PDFretrieved 6 October 2026 · published 1 October 2026
arXivprimary source
HTML full text — Groundability, Not Scale Alone
Original Archived PDFretrieved 6 October 2026 · published 1 October 2026
EU AI Act Explorersecondary
Human oversight — Article 14
Original Archived PDFretrieved 6 October 2026 · published 2 August 2026
EUR-Lexprimary source
Regulation (EU) 2024-2847 — Cyber Resilience Act
Original Archived PDFretrieved 6 October 2026 · published 20 November 2024
EUR-Lexprimary source
Regulation (EU) 2026-1744 — Digital Omnibus on AI
Original Archived PDFretrieved 6 October 2026 · published 27 July 2026

Every source for this edition, as one PDF BibTeX — all editions

Corrections

What changed after publication

None this edition. One identifier and one date were checked and held: arXiv 2610.01023 matches the story (2610 = October 2026) and the 1 October submission is the event date, against the 2 October cs.SE announcement that a coverage-date reading would have used. The arXiv PDF front matter could not be read (fetch proxy returned HTTP 429 and was not retried), so affiliations were taken from the work's own HTML rendering and are stated collectively rather than per author.