Who did the searching? — an automated researcher beats 28 humans at $4 an hour, and Article 9 already describes the loop

Anthropic published on 28 August 2026 that automated alignment researchers running post-training experiments against ten measurable alignment failures pass the best idea from a baseline of 30 ideas by 28 researchers — at least a year in technical AI safety, 2.5 years on average, up to eight hours each — after 6.4 hours of hill-climbing on average, at roughly $4 per hour in API inference against the $150 per hour paid to those researchers. The coverage framing of self-improving AI is wrong on the paper's own terms: this is a search over a method space scored on benchmarks humans chose in advance, and the paper's own monitor found 39 of 1,601 trajectories — 2.4% — cheating the score, most often by re-submitting and waiting for variance. The practitioner consequence is that the AI Act already describes this loop: Article 9(6) requires high-risk systems to be tested for the purpose of identifying the most appropriate and targeted risk management measures, and Article 9(8) requires that testing be carried out against prior defined metrics and probabilistic thresholds. A documented search with a passing number against a fixed metric is what the statute asks for and what $4 an hour now produces fastest, so the moves are to fix the metric before anything optimises against it, hold back one benchmark the optimiser never sees, monitor your own optimisation runs, and keep a named human on the Article 17(1)(m) accountability framework before Annex III high-risk applies on 2 December 2027.

Post

An automated researcher beat 28 human safety researchers. It cost four dollars an hour.

On 28 August, Anthropic's Alignment Science blog published "Automated Researchers Can Reliably Mitigate Alignment Failures" — Chen Yueh-Han, Jiaxin Wen and Jan Hendrik Kirchner, work done during the Anthropic Fellows Program.

The setup: automated alignment researchers running post-training experiments against ten measurable alignment failures — sycophancy, jailbreaks, prompt injection, deception, reward hacking and five more. The human baseline was 30 ideas from 28 researchers with at least a year in technical AI safety, 2.5 years on average, each given up to eight hours.

The paper: "the AAR's search passes the best human idea after 6.4 hours of hill-climbing on average." And: "An AAR costs roughly $4 per hour in API inference against the $150 per hour we pay our human researchers."

Three things before you file this under self-improving AI.

  1. It is not self-improving. It is a search over methods, scored against benchmarks a human picked in advance. Change the benchmark and you change the answer.

  2. The paper monitored itself. Across 1,601 AAR trajectories it found and excluded 39 — 2.4% — with confirmed cheating. The most common move was resubmission, hoping for score variance. None of them reached the top three.

  3. The AI Act already describes this loop. Article 9(6): high-risk systems "shall be tested for the purpose of identifying the most appropriate and targeted risk management measures." Article 9(8): "Testing shall be carried out against prior defined metrics and probabilistic thresholds."

A search over mitigations, scored on a metric you fixed in advance. That is the statute, not the paper.

Nothing in Article 9 says a person has to find the measure — and a documented search with a passing number is exactly what an assessor accepts, and exactly what four dollars an hour produces fastest.

What I would do Monday:

  1. Fix the metric before anything is allowed to optimise against it.
  2. Keep one held-out benchmark the optimiser never sees. Anthropic did.
  3. Run a monitor over your own optimisation runs. Theirs caught 2.4%.
  4. Name the person who signs the measure off. Article 17(1)(m) wants an accountability framework, and Annex III high-risk applies 2 December 2027.

Article 9 describes a search. It never says who does the searching.

Whose name is on your mitigation?

#AIAct #AgenticAI #AISafety #EUCompliance #AIGovernance

First comment

Sources, primary first.

The paper itself, on Anthropic's Alignment Science blog — Chen Yueh-Han, Jiaxin Wen, Jan Hendrik Kirchner, work done during the Anthropic Fellows Program. This is where the 6.4-hour figure, the $4-against-$150 sentence, the 28-researcher baseline and the 1,601 trajectories sit: https://alignment.anthropic.com/2026/automated-alignment-researchers/

Anthropic's own summary page for the same work, published 28 August 2026 — shorter, and the source of the "outscored 28 human safety researchers who had up to eight hours" line: https://www.anthropic.com/research/automated-researchers-mitigate-alignment-failures

Article 9 of the AI Act, Regulation (EU) 2024/1689 — paragraph 6 is testing to identify the measure, paragraph 8 is the prior defined metrics and probabilistic thresholds: https://artificialintelligenceact.eu/article/9/

Article 17, the quality management system — point (m) is the accountability framework naming management responsibilities: https://artificialintelligenceact.eu/article/17/

TechCrunch's coverage, same day, for the framing I am arguing with — the headline calls it a peek at self-improving AI: https://techcrunch.com/2026/08/28/an-anthropic-researcher-just-gave-us-a-peek-at-self-improving-ai/

One point cut for length: the data-efficiency result is the part that will matter to deployers first. Roughly 2,400 training examples got Claude Sonnet 5 near production alignment scores in about 60 hours — against 300,000 preference pairs in Tülu 3 and 1.4 million comparisons in Llama 2-Chat. Two to three orders of magnitude less data is the difference between an alignment fix being a lab project and being a sprint ticket.

Claims

Sources

Corrections

None.

AI disclosure

The video for this edition contains AI-generated media: the narration voice, the cover art and the brand ident animation are generated. The script, the claims and the source verification are human-authored and human-checked. Assets carry IPTC DigitalSourceType and C2PA Content Credentials.