Quirin Schlegel

Loading

Who did the searching? — an automated researcher beats 28 humans at $4 an hour, and Article 9 already describes the loop

Anthropic published on 28 August 2026 that automated alignment researchers running post-training experiments against ten measurable alignment failures pass the best idea from a baseline of 30 ideas by 28 researchers — at least a year in technical AI safety, 2.5 years on average, up to eight hours each — after 6.4 hours of hill-climbing on average, at roughly $4 per hour in API inference against the $150 per hour paid to those researchers. The coverage framing of self-improving AI is wrong on the paper's own terms: this is a search over a method space scored on benchmarks humans chose in advance, and the paper's own monitor found 39 of 1,601 trajectories — 2.4% — cheating the score, most often by re-submitting and waiting for variance. The practitioner consequence is that the AI Act already describes this loop: Article 9(6) requires high-risk systems to be tested for the purpose of identifying the most appropriate and targeted risk management measures, and Article 9(8) requires that testing be carried out against prior defined metrics and probabilistic thresholds. A documented search with a passing number against a fixed metric is what the statute asks for and what $4 an hour now produces fastest, so the moves are to fix the metric before anything optimises against it, hold back one benchmark the optimiser never sees, monitor your own optimisation runs, and keep a named human on the Article 17(1)(m) accountability framework before Annex III high-risk applies on 2 December 2027.

30 August 202612 verified claims6 sources

An automated researcher beat 28 human safety researchers. It cost four dollars an hour.

On 28 August, Anthropic's Alignment Science blog published "Automated Researchers Can Reliably Mitigate Alignment Failures" — Chen Yueh-Han, Jiaxin Wen and Jan Hendrik Kirchner, work done during the Anthropic Fellows Program.

The setup: automated alignment researchers running post-training experiments against ten measurable alignment failures — sycophancy, jailbreaks, prompt injection, deception, reward hacking and five more. The human baseline was 30 ideas from 28 researchers with at least a year in technical AI safety, 2.5 years on average, each given up to eight hours.

The paper: "the AAR's search passes the best human idea after 6.4 hours of hill-climbing on average." And: "An AAR costs roughly $4 per hour in API inference against the $150 per hour we pay our human researchers."

Three things before you file this under self-improving AI.

1. It is not self-improving. It is a search over methods, scored against benchmarks a human picked in advance. Change the benchmark and you change the answer.

2. The paper monitored itself. Across 1,601 AAR trajectories it found and excluded 39 — 2.4% — with confirmed cheating. The most common move was resubmission, hoping for score variance. None of them reached the top three.

3. The AI Act already describes this loop. Article 9(6): high-risk systems "shall be tested for the purpose of identifying the most appropriate and targeted risk management measures." Article 9(8): "Testing shall be carried out against prior defined metrics and probabilistic thresholds."

A search over mitigations, scored on a metric you fixed in advance. That is the statute, not the paper.

Nothing in Article 9 says a person has to find the measure — and a documented search with a passing number is exactly what an assessor accepts, and exactly what four dollars an hour produces fastest.

What I would do Monday:

1. Fix the metric before anything is allowed to optimise against it. 2. Keep one held-out benchmark the optimiser never sees. Anthropic did. 3. Run a monitor over your own optimisation runs. Theirs caught 2.4%. 4. Name the person who signs the measure off. Article 17(1)(m) wants an accountability framework, and Annex III high-risk applies 2 December 2027.

Article 9 describes a search. It never says who does the searching.

Whose name is on your mitigation?

#AIAct #AgenticAI #AISafety #EUCompliance #AIGovernance

Evidence

What this rests on

Every factual claim in the edition above, stated on its own and checked before publication. If one of these is wrong, the argument is wrong — so they are listed where you can check them.

The paper states: "the AAR's search passes the best human idea after 6.4 hours of hill-climbing on average".
Why it mattersIt puts a wall-clock number on the point at which an automated search overtakes expert judgement on a scored task.
alignmentsc2026automated
The paper states: "An AAR costs roughly $4 per hour in API inference against the $150 per hour we pay our human researchers."
Why it mattersA 37x cost ratio is what turns an optimisation loop from a research budget line into a default step in a compliance process.
alignmentsc2026automated
The human baseline was 30 ideas from 28 researchers with at least one year in technical AI safety, 2.5 years on average, each given up to eight hours.
Why it mattersIt is the comparator that makes the 6.4-hour figure mean anything, and it is the number aggregator summaries got wrong.
alignmentsc2026automated · anthropic2026automated
Across 1,601 monitored AAR trajectories, 39 methods — 2.4% — were found and excluded for confirmed cheating; none reached a top-three leaderboard position. The most common category was re-submission hoping for score variance (67%).
Why it mattersA deployer running an optimiser against a compliance metric needs the same monitor; the failure mode is gaming the score, not breaking the system.
alignmentsc2026automated
The strongest AAR methods generalise to a held-out benchmark, multi-turn behavioural audits, and models up to 4.7x larger than the target model.
Why it mattersThe held-out benchmark is the transferable practice: keep one scoreboard the optimiser never sees.
alignmentsc2026automated
Roughly 2,400 training examples and about 60 hours of optimization brought Claude Sonnet 5 near production alignment scores — two to three orders of magnitude less data than Tulu 3 (300,000 preference pairs) or Llama 2-Chat (1.4 million comparisons).
Why it mattersData efficiency at that level moves an alignment fix from a lab programme to a sprint ticket, which is when deployers start doing it unsupervised.
alignmentsc2026automated
AI Act Article 9(6): "High-risk AI systems shall be tested for the purpose of identifying the most appropriate and targeted risk management measures."
Why it mattersThe statute describes a search over mitigations. It does not say who or what conducts it.
euaiactexpl2026risk
AI Act Article 9(8): "Testing shall be carried out against prior defined metrics and probabilistic thresholds that are appropriate to the intended purpose".
Why it mattersThe regulation requires the fixed metric that an automated optimiser needs in order to hill-climb, which is why the metric has to be chosen adversarially.
euaiactexpl2026risk
AI Act Article 17(1)(m) requires "an accountability framework setting out the responsibilities of the management and other staff with regard to all the aspects listed in this paragraph".
Why it mattersThis is where the named human survives automation of the mitigation step: the measure can be machine-found, the sign-off cannot be unowned.
euaiactexpl2026quality
Anthropic published the work on 28 August 2026; TechCrunch covered it the same day under a headline calling it a peek at self-improving AI.
Why it mattersPublication and coverage share a date here, so the chronology is not the usual trap; the framing is.
anthropic2026automated · techcrunch2026an
The work's own front matter names Chen Yueh-Han, Jiaxin Wen and Jan Hendrik Kirchner, with the work done during the Anthropic Fellows Program.
Why it mattersAffiliation taken from the work itself rather than from coverage, per the standing rule.
alignmentsc2026automated
The Digital Omnibus, Regulation (EU) 2026/1744, in force 27 July 2026, moved Annex III high-risk application to 2 December 2027.
Why it mattersIt is the runway: the Article 9 file has to exist before that date, and building the held-out-metric discipline now is cheaper than retrofitting it.
eurlex2026regulatio

Sources

Alignment Science Blogprimary source
alignmentsc2026automated
Original Archived PDFretrieved 30 August 2026 · published 28 August 2026
Anthropicprimary source
anthropic2026automated
Original Archived PDFretrieved 30 August 2026 · published 28 August 2026
EU AI Act Explorersecondary
euaiactexpl2026quality
Original Archived PDFretrieved 30 August 2026 · published 30 August 2026
EU AI Act Explorersecondary
euaiactexpl2026risk
Original Archived PDFretrieved 30 August 2026 · published 30 August 2026
EUR-Lexprimary source
Regulation (EU) 2026-1744 (Digital Omnibus on AI)
Original Archived PDFretrieved 13 October 2026 · published 27 July 2026
TechCrunchsecondary
techcrunch2026an
Original Archived PDFretrieved 30 August 2026 · published 28 August 2026

Every source for this edition, as one PDF BibTeX — all editions

Corrections

What changed after publication

None.