Quirin Schlegel

Loading

It removes the reader — code review agents reach 71.9% of bugs in their top five comments, and the German act that makes that record discoverable lands in 56 days

CRBugBench puts the best code review agent at 71.9% of human-verified bugs inside its top five comments, up from 47.7% five months earlier — but top-five is a recall measure that presumes a reader, and the score degrades on complex changes, on bugs needing reasoning beyond the local code, and on proprietary repositories. The regulatory reflex is AI Act Article 14, which is high-risk only and deferred to 2 December 2027 for Annex III. The live clock is German: the Bundestag passed the Gesetz zur Modernisierung des Produkthaftungsrechts on 8 October 2026, applicable 9 December 2026, making software a product (Paragraph 2(1) Nr. 3), letting a court order disclosure of evidence including from third parties (Paragraph 19) and presuming the defect on non-disclosure (Paragraph 20(1)). A lights-off factory stops producing readers, not findings — and the findings are the record.

14 October 202612 verified claims5 sources

A pull request with five review comments on it. Nobody resolved any of them, and it merged anyway.

A paper posted to arXiv on 9 October asks a question a lot of teams have already answered by accident: should a software factory run with the lights off?

Flat:

1. CRBugBench pairs human-verified bugs with the pull request that introduced them: 1,000 review tasks across 98 open-source GitHub repositories, plus 800 from one large company's proprietary code. The best agent — claude-opus-5 driven by copilot-cli — catches 71.9% of bugs within its top five comments, against 47.7% for opus-4.6 five months earlier.

2. Read the metric before you quote it. Top-five means the bug appears somewhere in five comments. Something still has to pick which one, so 71.9% in five is a triage aid, not a gate.

3. Where it degrades matters more: complex changes, bugs needing reasoning beyond the local code, and proprietary repositories — the only ones you actually have. The lever is prompt design, not compute: exploration-nudging gains up to 5.8% for roughly 10% more cost.

4. Now the European part, and it is not the AI Act. Article 14's human-oversight duty is high-risk only and sits at 2 December 2027 for Annex III after the Digital Omnibus. What is 56 days out is German. The Bundestag passed the Gesetz zur Modernisierung des Produkthaftungsrechts on 8 October, implementing Directive (EU) 2024/2853, applicable from 9 December 2026. § 2(1) Nr. 3 makes software a product. § 19 lets a court order you to disclose evidence once a claim is plausible — the Rechtsausschuss extended that to third parties. § 20(1) presumes the defect if you fail to disclose, if the product misses mandatory safety requirements, or if an obvious malfunction caused the harm. § 23 leaves anything on the market by 8 December under the old law.

A lights-off factory does not stop producing findings. It stops producing readers. Every comment your agent filed records what your pipeline knew — and from December a court can ask to see it.

My Monday: take the last twenty pull requests an agent reviewed and you merged. Read every comment nobody resolved. Count the ones that named a real defect. That number is not a quality metric. It is your disclosure exposure.

A comment nobody read is not a review.

Which of your agent's comments went unanswered?

#AgenticAI #EUAIAct #CodingAgents #Produkthaftung #SoftwareEngineering

AI disclosure: the narration voice, the cover art and the brand ident animation in this video are AI-generated. The script, the claims and the source checks are mine.

Evidence

What this rests on

Every factual claim in the edition above, stated on its own and checked before publication. If one of these is wrong, the argument is wrong — so they are listed where you can check them.

CRBugBench mines human-verified bugs from repository history and pairs each with the pull request that introduced it, comprising 1,000 review tasks across 98 open-source GitHub repositories plus 800 tasks from proprietary repositories at one large company.
Why it mattersEight hundred of the tasks come from one company's own code. That split is the only part of the benchmark that resembles your repository.
arxiv2026when
On CRBugBench the best agent, claude-opus-5 with copilot-cli, catches 71.9% of bugs within its top five comments, up from 47.7% for opus-4.6 over five months.
Why it mattersA twenty-four-point jump in five months is the number that will carry a slide. It is also the number that makes a procurement decision look safe for longer than it is.
arxiv2026when
The CRBugBench headline figure is a top-five measure over the agent's comments rather than a single verdict, and the paper reports agents weaker on complex changes, on bugs requiring reasoning beyond the local code, and noticeably worse on proprietary repositories where project context matters more.
Why it mattersTop-five is a recall measure, so the figure presumes a reader choosing among five comments. Quote it as a gate and you have assumed away the person the metric depends on.
arxiv2026when
The paper reports that scaling compute helps most through prompt design, with exploration-nudging prompts improving performance by up to 5.8% at 10% higher cost.
Why it mattersThe cheapest lever is the one you own. Prompting beats buying compute here, which also means the result is not reproducible across teams that prompt differently.
arxiv2026when
The paper's own adoption guidance is to scale review effort to pull-request complexity, steer agents toward project priorities, provide repository knowledge, choose models by pull-request characteristics and monitor quality as models change.
Why it mattersThe paper most likely to be cited for automation argues for supervision. Read the recommendations before quoting the headline.
arxiv2026when
The Bundestag passed the Gesetz zur Modernisierung des Produkthaftungsrechts on Thursday 8 October 2026, on the Beschlussempfehlung of the Ausschuss fuer Recht und Verbraucherschutz dated 7 October 2026 (Drucksache 21/8429), and under Art. 17(1) the act takes effect on 9 December 2026.
Why it mattersFifty-six days. The clock is national, not the AI Act's, and it is the nearest one on the calendar for anyone shipping software from Germany.
deutscherbu2026erste · deutscherbu2026beschluss
Paragraph 2(1) Nr. 3 of the new German product liability act includes software in the definition of product, excluding free and open-source software developed or supplied outside a business activity, and Paragraph 7(3) requires considering a product's ability to learn or acquire new functions after it is placed on the market.
Why it mattersSoftware being a product is what puts a code-review pipeline inside the liability frame at all. Paragraph 7(3) then stops 'it changed after release' from being a defence.
deutscherbu2026beschluss
Paragraph 19 lets a court order disclosure of evidence on a plausible claim, and the committee amendment extends this to evidence held by third parties, with the ZPO right to refuse testimony applying and separate safeguards for trade secrets.
Why it mattersDisclosure reaching third parties means your vendor's review logs are not out of reach either. The record does not have to be yours to be produced.
deutscherbu2026beschluss
Paragraph 20(1) presumes a defect where the defendant fails to disclose evidence despite a court order, where the product does not meet mandatory safety requirements, or where an obvious malfunction caused the harm; Paragraph 20 was unchanged by the committee.
Why it mattersThree independent triggers, any one of which flips the burden. Failure to disclose is the one a lights-off pipeline can walk into without noticing.
deutscherbu2026beschluss
Under Paragraph 23, products placed on the market by 8 December 2026 remain subject to the old Produkthaftungsgesetz.
Why it mattersThe cut-off is a reason to know exactly what you shipped before 8 December, not a reason to relax.
deutscherbu2026beschluss
AI Act Article 14's human-oversight duty applies to high-risk systems only, from 2 December 2027 for Annex III and 2 August 2028 for Annex I after Regulation (EU) 2026/1744.
Why it mattersThe duty everyone reaches for is not in force and does not reach a coding agent. Saying so is the difference between a practitioner's read and a compliance newsletter.
euaiactexpl2026human
Art. 22(1) of Directive (EU) 2024/2853 sets the transposition deadline at 9 December 2026, which the German act's entry into force matches.
Why it mattersThe German date is not a national invention; it is the Directive's own deadline, so the same clock runs in every member state.
deutscherbu2026beschluss · eurlex2024directive

Sources

arXivprimary source
arxiv2026when
Original Archived PDFretrieved 14 October 2026 · published 9 October 2026
Deutscher Bundestagprimary source
deutscherbu2026beschluss
Original Archived PDFretrieved 14 October 2026 · published 7 October 2026
Deutscher Bundestag Textarchivprimary source
deutscherbu2026erste
Original Archived PDFretrieved 14 October 2026 · published 4 March 2026
EU AI Act Explorersecondary
euaiactexpl2026human
Original Archived PDFretrieved 14 October 2026 · published 2 August 2026
EUR-Lexprimary source
eurlex2024directive
Original Archived PDFretrieved 14 October 2026 · published 23 October 2024

Every source for this edition, as one PDF BibTeX — all editions

Corrections

What changed after publication

None.