It removes the reader — code review agents reach 71.9% of bugs in their top five comments, and the German act that makes that record discoverable lands in 56 days

CRBugBench puts the best code review agent at 71.9% of human-verified bugs inside its top five comments, up from 47.7% five months earlier — but top-five is a recall measure that presumes a reader, and the score degrades on complex changes, on bugs needing reasoning beyond the local code, and on proprietary repositories. The regulatory reflex is AI Act Article 14, which is high-risk only and deferred to 2 December 2027 for Annex III. The live clock is German: the Bundestag passed the Gesetz zur Modernisierung des Produkthaftungsrechts on 8 October 2026, applicable 9 December 2026, making software a product (Paragraph 2(1) Nr. 3), letting a court order disclosure of evidence including from third parties (Paragraph 19) and presuming the defect on non-disclosure (Paragraph 20(1)). A lights-off factory stops producing readers, not findings — and the findings are the record.

Post

A pull request with five review comments on it. Nobody resolved any of them, and it merged anyway.

A paper posted to arXiv on 9 October asks a question a lot of teams have already answered by accident: should a software factory run with the lights off?

Flat:

CRBugBench pairs human-verified bugs with the pull request that introduced them: 1,000 review tasks across 98 open-source GitHub repositories, plus 800 from one large company's proprietary code. The best agent — claude-opus-5 driven by copilot-cli — catches 71.9% of bugs within its top five comments, against 47.7% for opus-4.6 five months earlier.
Read the metric before you quote it. Top-five means the bug appears somewhere in five comments. Something still has to pick which one, so 71.9% in five is a triage aid, not a gate.
Where it degrades matters more: complex changes, bugs needing reasoning beyond the local code, and proprietary repositories — the only ones you actually have. The lever is prompt design, not compute: exploration-nudging gains up to 5.8% for roughly 10% more cost.
Now the European part, and it is not the AI Act. Article 14's human-oversight duty is high-risk only and sits at 2 December 2027 for Annex III after the Digital Omnibus. What is 56 days out is German. The Bundestag passed the Gesetz zur Modernisierung des Produkthaftungsrechts on 8 October, implementing Directive (EU) 2024/2853, applicable from 9 December 2026. § 2(1) Nr. 3 makes software a product. § 19 lets a court order you to disclose evidence once a claim is plausible — the Rechtsausschuss extended that to third parties. § 20(1) presumes the defect if you fail to disclose, if the product misses mandatory safety requirements, or if an obvious malfunction caused the harm. § 23 leaves anything on the market by 8 December under the old law.

A lights-off factory does not stop producing findings. It stops producing readers. Every comment your agent filed records what your pipeline knew — and from December a court can ask to see it.

My Monday: take the last twenty pull requests an agent reviewed and you merged. Read every comment nobody resolved. Count the ones that named a real defect. That number is not a quality metric. It is your disclosure exposure.

A comment nobody read is not a review.

Which of your agent's comments went unanswered?

#AgenticAI #EUAIAct #CodingAgents #Produkthaftung #SoftwareEngineering

AI disclosure: the narration voice, the cover art and the brand ident animation in this video are AI-generated. The script, the claims and the source checks are mine.

First comment

Sources, the paper first.

(When) Should a Software Factory be Lights-off? Benchmarking Code Review Agents on Bug Detection — arXiv 2610.13668, v1 submitted 9 October 2026, announced in the 12 October listing. Every figure above is from the abstract record. The HTML full text returned HTTP 429 to my fetch proxy and was not retried, so no per-repository or per-complexity breakdown is claimed beyond the direction the abstract states, and no author affiliation is given because the abstract page shows none: https://arxiv.org/abs/2610.13668

Deutscher Bundestag, Drucksache 21/8429 of 7 October 2026 — Beschlussempfehlung und Bericht of the Ausschuss für Recht und Verbraucherschutz on Drucksache 21/4297. This is where § 19 on disclosure, § 20(1) on the presumption, § 2(1) Nr. 3 on software, § 7(3) on post-market learning and § 23 on the cut-off are to be read, together with the committee's amendment extending disclosure to third parties and the trade-secret safeguards: https://dserver.bundestag.de/btd/21/084/2108429.pdf

Deutscher Bundestag textarchiv — the passage date. First reading 4 March 2026, adopted by the Bundestag on Thursday 8 October 2026: https://www.bundestag.de/dokumente/textarchiv/2026/kw10-de-produkthaftungsrecht-1150448

Directive (EU) 2024/2853 on liability for defective products, the instrument being implemented — Article 22(1) sets the 9 December 2026 transposition deadline that the German act's entry into force matches: https://eur-lex.europa.eu/eli/dir/2024/2853/oj/eng

AI Act Article 14, human oversight — the duty the reflex reaches for, with the application dates at the foot of the page: 2 December 2027 for Annex III, 2 August 2028 for Annex I: https://artificialintelligenceact.eu/article/14/

One point cut for length. The paper's own recommendation is not lights-off. It is to scale review effort to pull-request complexity, feed the agent repository knowledge, pick the model per pull request and keep monitoring as models change — four pieces of advice that all assume somebody is still in the loop. The benchmark that gets quoted as a case for automation was written by people arguing for supervision.

AI disclosure: the narration voice, the cover art and the brand ident animation in this video are AI-generated. The script, the claims and the source checks are mine.

Claims

Sources

Corrections

None.

AI disclosure

The video for this edition contains AI-generated media: the narration is a synthetic voice, the cover art and any background imagery are generated, and the brand ident animation is generated. The script, the claims and the source verification are human-authored and human-checked. Assets carry IPTC DigitalSourceType and C2PA Content Credentials.

It removes the reader — code review agents reach 71.9% of bugs in their top five comments, and the German act that makes that record discoverable lands in 56 days

CRBugBench puts the best code review agent at 71.9% of human-verified bugs inside its top five comments, up from 47.7% five months earlier — but top-five is a recall measure that presumes a reader, and the score degrades on complex changes, on bugs needing reasoning beyond the local code, and on proprietary repositories. The regulatory reflex is AI Act Article 14, which is high-risk only and deferred to 2 December 2027 for Annex III. The live clock is German: the Bundestag passed the Gesetz zur Modernisierung des Produkthaftungsrechts on 8 October 2026, applicable 9 December 2026, making software a product (Paragraph 2(1) Nr. 3), letting a court order disclosure of evidence including from third parties (Paragraph 19) and presuming the defect on non-disclosure (Paragraph 20(1)). A lights-off factory stops producing readers, not findings — and the findings are the record.

Post

A pull request with five review comments on it. Nobody resolved any of them, and it merged anyway.

A paper posted to arXiv on 9 October asks a question a lot of teams have already answered by accident: should a software factory run with the lights off?

Flat:

CRBugBench pairs human-verified bugs with the pull request that introduced them: 1,000 review tasks across 98 open-source GitHub repositories, plus 800 from one large company's proprietary code. The best agent — claude-opus-5 driven by copilot-cli — catches 71.9% of bugs within its top five comments, against 47.7% for opus-4.6 five months earlier.
Read the metric before you quote it. Top-five means the bug appears somewhere in five comments. Something still has to pick which one, so 71.9% in five is a triage aid, not a gate.
Where it degrades matters more: complex changes, bugs needing reasoning beyond the local code, and proprietary repositories — the only ones you actually have. The lever is prompt design, not compute: exploration-nudging gains up to 5.8% for roughly 10% more cost.
Now the European part, and it is not the AI Act. Article 14's human-oversight duty is high-risk only and sits at 2 December 2027 for Annex III after the Digital Omnibus. What is 56 days out is German. The Bundestag passed the Gesetz zur Modernisierung des Produkthaftungsrechts on 8 October, implementing Directive (EU) 2024/2853, applicable from 9 December 2026. § 2(1) Nr. 3 makes software a product. § 19 lets a court order you to disclose evidence once a claim is plausible — the Rechtsausschuss extended that to third parties. § 20(1) presumes the defect if you fail to disclose, if the product misses mandatory safety requirements, or if an obvious malfunction caused the harm. § 23 leaves anything on the market by 8 December under the old law.

A lights-off factory does not stop producing findings. It stops producing readers. Every comment your agent filed records what your pipeline knew — and from December a court can ask to see it.

A comment nobody read is not a review.

Which of your agent's comments went unanswered?

#AgenticAI #EUAIAct #CodingAgents #Produkthaftung #SoftwareEngineering

AI disclosure: the narration voice, the cover art and the brand ident animation in this video are AI-generated. The script, the claims and the source checks are mine.

First comment

Sources, the paper first.

AI disclosure: the narration voice, the cover art and the brand ident animation in this video are AI-generated. The script, the claims and the source checks are mine.