The reviewer was big enough — a weak model can audit a strong coding agent when the evidence is checkable, and the Article 14 that binds today is not the one you reach for

A weaker reviewer can audit a stronger coding agent, but only when the evidence it is handed has been independently checked: across a ladder from 8B to 235B parameters, reviewer size is not a consistent predictor of quality, and with official execution evidence two of six reviewers classified all 122 held-out traces correctly. Without official tests - the deployment case - the paper's frozen cascade separates catch from over-rejection by only ten to thirteen points, which is not a gate. The regulatory reflex is the wrong Article 14: AI Act Article 14 human oversight moved to 2 December 2027 with Annex III under Regulation (EU) 2026/1744, while Cyber Resilience Act Article 14 has bound manufacturers since 11 September 2026 and keys its final-report clock to the availability of a corrective measure rather than to the incident.

Post

A diff in the right file. A summary that sounds certain. A test suite that never asked the question the ticket was about.

That patch gets approved, and the approval means nothing.

On 1 October, researchers at UC Berkeley and Virginia Tech asked when a weaker model can reliably decide whether a coding agent's patch solves its issue. 411 execution-labelled traces from three agents, 101 controlled cases, six reviewer models from Llama-3.1-8B to Qwen3-235B.

Flat:

  1. Hand the reviewer official execution evidence and five of six improve on both measures; two classified all 122 held-out traces correctly. Hand them structured-but-unchecked evidence — selected, confident, nothing verified — and both numbers move the wrong way together: more defects caught, more good patches rejected.

  2. Before you conclude the fix is a bigger reviewer: reviewer size is not a consistent predictor of quality. The ladder spans 8B to 235B and the ranking is not monotonic. The lever is groundability — whether an independent check signed the evidence — not scale.

  3. Official tests don't exist in deployment, so the authors froze a cascade: static errors plus generated tests that first fail on the unpatched repo. On held-out sets, catch 0.76 against over-rejection 0.66 on GPT-5.4, catch 0.80 against 0.67 on Gemini. Ten to thirteen points of separation. That is not a gate.

  4. Now the European part, and it is not the article you expect. The oversight duty everyone reaches for is AI Act Article 14 — understand the system's limits well enough to spot anomalies. It moved to 2 December 2027 with Annex III, under Regulation (EU) 2026/1744, the Digital Omnibus. The Cyber Resilience Act has an Article 14 too, and it has bound manufacturers since 11 September 2026: 24 hours to warn on an actively exploited vulnerability, 72 to notify, a final report 14 days after a corrective measure is available. That clock starts at your fix, not your incident.

My Monday: make the gate produce a test that fails on the unpatched repo and passes after. Freeze one evidence format per gate. Track over-rejection next to catch — a gate nobody trusts gets switched off. And record which check signed the approval.

Oversight is not someone senior reading more carefully. It is a question your tooling can answer.

Which of your agent's approvals rests on a check that could have said no?

#AgenticAI #EUAIAct #CyberResilienceAct #CodeReview #AIGovernance

First comment

Sources, the paper first.

Groundability, Not Scale Alone — arXiv 2610.01023, submitted 1 October 2026. The 411 traces, the six reviewers, and the finding that reviewer size is not a consistent predictor: https://arxiv.org/abs/2610.01023

The full text as HTML. This is where the definitions live, and they matter: catch is P(reject given defect), over-rejection is P(reject given acceptable), coverage is the share of traces that get a decision at all: https://arxiv.org/html/2610.01023v1

The European Commission's own page on CRA reporting — who owes it, from when, and the 24-hour / 72-hour / 14-day structure. Note that open-source stewards are not in scope until 11 December 2027: https://digital-strategy.ec.europa.eu/en/policies/cra-reporting

Regulation (EU) 2024/2847, the Cyber Resilience Act itself, for the operative wording of the reporting duty: https://eur-lex.europa.eu/eli/reg/2024/2847/oj/eng

AI Act Article 14 — read 14(4)(a), on understanding the system's capacities and limitations well enough to detect anomalies. That is the sentence the paper is really about: https://artificialintelligenceact.eu/article/14/

Regulation (EU) 2026/1744, the Digital Omnibus on AI, in force 27 July 2026 — the instrument that moved Annex III and Article 14 with it: https://eur-lex.europa.eu/eli/reg/2026/1744/oj

The CRA timeline in one place, if you only want the dates: https://www.cyberresilienceact.eu/explained.html

One point I cut for length: the paper notes most false rejections happen when unresolved cases reach the reviewer at all — so some of that over-rejection is really an upstream routing problem, not a review problem. Which is its own argument for not letting the agent decide when it is done.

Claims

Sources

Corrections

None this edition. One identifier and one date were checked and held: arXiv 2610.01023 matches the story (2610 = October 2026) and the 1 October submission is the event date, against the 2 October cs.SE announcement that a coverage-date reading would have used. The arXiv PDF front matter could not be read (fetch proxy returned HTTP 429 and was not retried), so affiliations were taken from the work's own HTML rendering and are stated collectively rather than per author.

AI disclosure

The video for this edition contains AI-generated media: the narration is a synthetic voice, the cover art and any background imagery are generated, and the brand ident animation is generated. The script, the claims and the source verification are human-authored and human-checked. Assets carry IPTC DigitalSourceType and C2PA Content Credentials.