Quirin Schlegel

Loading

The gauge nobody asked for — Anthropic measured how much of its own model-building Claude leads, and Recital 110 named that risk without a number

Anthropic published three self-measured indexes of its own model-building this week: Claude "leads" 26% of its AI R&D work as of August 2026 against under 1% in February, about 30,000 agents run at any one time on its most-used internal platform, and of over a billion agent decisions in August 0.002% - about 1 in 47,000 - were blocked by the real-time monitor, with 100% of actions passing through it before execution. The obvious reading, that "leads" means unsupervised, is wrong on the lab's own terms: "leads" is AL4 on a scale proposed by Epoch AI, where the model completes most of a task end-to-end from a high-level prompt while a human supervises, and Anthropic states Claude is not operating fully autonomously for any measured subset. The second correction is the one every summary dropped: the ratings come from a Claude judge whose exact agreement with the humans who own the work is 59%, and the appendix gives the human-versus-human figure in the same breath - 35%, with model and human within one level 97% of the time. The European point is that nobody asked for any of this: Recital 110 of the AI Act already lists "risks from models of making copies of themselves or 'self-replicating' or training other models" among systemic risks, and Article 55(1)(b) and (c) require assessment, mitigation and serious-incident reporting to the AI Office without attaching a metric to any of it. So the index is a template, not a filing: a vendor's self-measured number is not evidence in a deployer's own file, and the method is still free to copy. The practitioner move is to compute Anthropic's three - coverage, review latency, escalation rate - for one agent class of your own, starting with coverage, which will not come back at 100%.

19 September 202610 verified claims5 sources

Claude now leads 26% of Anthropic's own AI research and development work. The number everyone quoted as the flaw is the one that makes it credible.

Anthropic published three self-measured indexes this week: how much of its AI R&D is done by AI, how well its agents are overseen, and where its compute goes.

What it reports, flat:

1. Claude "leads" 26% of Anthropic's AI R&D work as of August 2026, against under 1% in February. "Leads" is AL4 on a scale Epoch AI proposed - the model completes most of a task end-to-end from a high-level prompt, while a human supervises. AL5, fully autonomous, is not reached on any measured subset. Anthropic says that itself.

2. Roughly 30,000 agents at any one time, on its most-used internal platform - and the measurement covers that platform only. Over a billion agent decisions in August; 0.002%, about 1 in 47,000, blocked by the real-time monitor. Offline monitors flag one to two transcripts per thousand, and about 50 a week reach a human.

3. The ratings come from a Claude judge, and model-versus-human exact agreement is 59%. Every write-up I read ran that as the weakness. The appendix gives the other number in the same breath: human-versus-human agreement is 35%. Within one level, model and human match 97% of the time.

4. Now the European part, which nobody required. Recital 110 of the AI Act already lists, among systemic risks, "risks from models of making copies of themselves or 'self-replicating' or training other models". Article 55(1)(b) makes providers of GPAI models with systemic risk assess and mitigate them; 55(1)(c) makes them report serious incidents to the AI Office. No provision attaches a number to any of it.

5. So this is a template, not a filing - and that cuts both ways. A vendor's self-measured index is not evidence in your own file. The method is still free to copy.

My Monday: take one agent class in your estate and compute Anthropic's three - coverage, the share of actions a monitor actually sees; review latency; escalation rate. Coverage is the one that will not come back at 100%.

A lab grading itself is still a lab grading itself. This one published the rate at which its own graders disagree.

Which of your agents' actions has nothing ever looked at?

#AIAct #AgenticAI #EUTech #AIGovernance #DevSecOps

AI disclosure: the narration voice, the cover art and the brand ident animation in this video are AI-generated. The script, the claims and the source checks are mine.

Evidence

What this rests on

Every factual claim in the edition above, stated on its own and checked before publication. If one of these is wrong, the argument is wrong — so they are listed where you can check them.

Claude "leads" 26% of Anthropic's AI R&D work as of August 2026, against under 1% in February 2026.
Why it mattersThe headline number for how fast model-building itself is automating, with the baseline in the same sentence.
anthropic2026measureme
Anthropic states that Claude is not operating fully autonomously (AL5) for any measured subset of AI R&D work, and that work at or above "AI collaborates" is above 90%.
Why it mattersFixes the misreading that 26% means unsupervised operation. It does not.
anthropic2026measureme
The AL0-AL5 automation scale was developed by Epoch AI (JS Denain, Joe Kwon, Anson Ho, 17 June 2026). AL4 "leads" means the model completes most of the task end-to-end from a high-level prompt while the human supervises.
Why it mattersTells you whose scale this is and what the level actually asserts, so you can apply it to your own estate.
anthropic2026measureme · epochai2026toward
Approximately 30,000 agents were doing research and engineering work at Anthropic at any one time in its most-used internal platform as of August 2026; the measurements cover that platform only.
Why it mattersSets the scope: one platform, a point-in-time count, not the whole company.
anthropic2026measureme
Over a billion agent decisions were analysed over August 2026; 0.002% (about 1 in 47,000) were blocked by the online monitor. 100% of these agents' actions pass through that monitor before execution.
Why it mattersThe escalation rate practitioners will quote. Coverage, not the block rate, is the number that transfers.
anthropic2026measureme
Offline monitors flag roughly one to two transcripts in every thousand, about 100,000 transcripts per week, of which approximately 50 per week are escalated to human review.
Why it mattersShows how much human attention a fleet this size actually consumes: about fifty items a week.
anthropic2026measureme
Model-versus-human exact agreement on the automation ratings was 59%; human-versus-human exact agreement was 35%; model and human ratings were within one level of each other 97% of the time.
Why it mattersThe measurement-quality number every summary reported without its comparator.
anthropic2026measureme
Recital 110 of the EU AI Act lists among systemic risks "risks from models of making copies of themselves or 'self-replicating' or training other models".
Why it mattersThe EU already named models training other models as a systemic risk, two years before anyone measured it.
euaiactexpl2026selfrepli
Article 55(1)(b) requires providers of GPAI models with systemic risk to assess and mitigate possible systemic risks at Union level; Article 55(1)(c) requires them to keep track of, document and report serious incidents to the AI Office without undue delay.
Why it mattersWhat a systemic-risk provider actually owes, and to whom - none of it expressed as a metric.
euaiactexpl2026systemicr
In the week of 13 to 20 July 2026, about 6% of the compute that went to AI R&D was allocated to safety, and about 12% of the compute that went to AI-driven AI R&D. Anthropic states the underlying workload labels are best-effort and not verified, and that one week is not enough to show a trend.
Why it mattersThe compute split, with the caveats that stop it being read as a trend.
anthropic2026measureme

Sources

Anthropicprimary source
August 2026 risk report (redacted)
Original Archived PDFretrieved 19 September 2026 · published 1 August 2026
Anthropicprimary source
Measurements for understanding the pace of AI development inside frontier labs
Original Archived PDFretrieved 19 September 2026 · published 18 September 2026
EU AI Act Explorersecondary
Self-replication and training other models (Recital 110)
Original Archived PDFretrieved 19 September 2026 · published 1 January 2026
EU AI Act Explorersecondary
Systemic-risk model obligations (Article 55)
Original Archived PDFretrieved 19 September 2026 · published 1 January 2026
Epoch AIprimary source
Toward an O*NET for AI R&D
Original Archived PDFretrieved 19 September 2026 · published 17 June 2026

Every source for this edition, as one PDF BibTeX — all editions

Corrections

What changed after publication

Publication date: the primary page carries no visible date. The site's own metadata gives 2026-09-18T00:02:35Z and first coverage appeared 18 September 2026; one aggregator dated it 17 September. The edition says "this week" rather than asserting a day.

Cross-reference checked, not assumed: the post states "We published all of these measurements in our recent risk report". The redacted August 2026 risk report was read; it carries the alignment assessments and METR's red-teaming but not the coverage/latency/escalation table. The first comment says so rather than repeating the claim.

Agreement rates: several write-ups reported the 59% model-versus-human figure as a weakness without the 35% human-versus-human figure printed beside it in the same appendix. Both are given here.

Cite key euaiactexpl2026systemicr reused for Article 55 rather than minted again; the same URL already carries two keys in the vault from 2026-08-13 and 2026-09-09.