Nothing asked, so nothing held — coding agents ship functional code nine times in ten and secure code one time in four, and the only AI Act date left this year is a label

The Last Secure Code Benchmark grades 450 tasks built from 150 real vulnerabilities with two hidden suites, one functional and one that replays the original proof-of-concept attack. Six frontier agents pass functionally 73.6% to 91.3% of the time and functionally-and-securely only 16.4% to 25.1%. The mechanism is not capability: for the top agent the security-only rate (25.3%) and the both-suites rate (25.1%) differ by two tenths of a point, so almost everything that survives the attack replay also works, and the specification simply never named the attack. The regulatory reflex is the AI Act, which does not reach code quality: the next AI Act date is 2 December 2026, Article 50(2) machine-readable marking of synthetic output, a provenance duty; Annex III human oversight moved to 2 December 2027 under Regulation (EU) 2026/1744; and the Cyber Resilience Act essential requirements that actually speak to attack surface apply 11 December 2027. Acceptance criteria are the only live control.

Post

Six frontier coding agents were given 450 real tasks. They got the code working between 74% and 91% of the time.

Both working and secure: 16% to 25%.

The Last Secure Code Benchmark (arXiv 2610.14766, submitted 10 October 2026) is built from 150 real vulnerabilities across seven languages and 24 CWEs. Every task specification asks only for functionality. None of them mentions security. Two hidden suites then grade the result: one checks the behaviour that was asked for, the other replays the original proof-of-concept attack against the running program. A task counts only if both pass. No partial credit, no judge model.

Four things I took from it:

The best agent, Claude Fable 5 in Claude Code, scored 91.1% functional and 25.1% functional-and-secure. The agent with the highest functional score of all, 91.3%, finished last on the combined measure at 16.4%. Passing more tests predicted nothing.
Security is the binding constraint, not capability. For the top agent the security pass rate (25.3%) and the both-suites rate (25.1%) differ by two tenths of a point. Nearly everything that survives the attack replay also works. The reverse is not remotely true.
The exposure does not come from autonomy. Averaged over the six agents, functional pass falls from about 93% when filling in a single function to about 80% when rebuilding a source tree, while functional-and-secure stays flat between 19% and 23%. It was already there at the smallest unit of work.
The best model solved only 12.0% of the 150 scenarios securely at all three granularities. For the other 88%, something broke somewhere.

The European reflex is to reach for the AI Act. It does not reach this. The next AI Act date is 2 December 2026, 48 days out, and it is Article 50(2): mark synthetic output so it is machine-readably detectable. A duty about provenance, not about the CWE inside. Annex III human oversight moved to 2 December 2027. The Cyber Resilience Act's essential requirements, the ones that actually say limit attack surfaces, apply 11 December 2027.

So nothing in force today asks whether your agent's code is secure. Your acceptance criteria are the only thing that does, and the finding here is that a specification which never names the attack buys no protection from it.

What is in your definition of done that an attacker would recognise?

#AgenticEngineering #AIAct #SecureCoding #EURegulation #AppSec

AI disclosure: the narration voice, the cover art and the brand ident animation in this video are AI-generated. The script, the claims and the source checks are mine.

First comment

Sources.

The paper, abstract and metadata record: https://arxiv.org/abs/2610.14766

Per-model leaderboard: functional, secure, both, and the all-three-granularities column: https://last-secure-code-benchmark.github.io/

Benchmark code, task construction and the grading harness: https://github.com/alibaba/last-secure-code-benchmark

AI Act Article 50 transparency obligations, including the machine-readable marking limb: https://artificialintelligenceact.eu/article/50/

Cyber Resilience Act timeline, separating the September 2026 reporting duty from the December 2027 essential requirements: https://digital-strategy.ec.europa.eu/en/policies/cyber-resilience-act

One point cut for length: the paper credits the work to teams at Alibaba Group and the National University of Singapore, and the arXiv abstract page lists no affiliations at all. Worth remembering when a benchmark puts a vendor's own model at the top of a board the vendor also publishes. Here it does not: the top entry is Claude Fable 5, not Qwen3.8-Max.

AI disclosure: the narration voice, the cover art and the brand ident animation in this video are AI-generated. The script, the claims and the source checks are mine.

Claims

Sources

Corrections

None for this edition.

AI disclosure

The video for this edition contains AI-generated media: the narration is a synthetic voice, the cover art and any background imagery are generated, and the brand ident animation is generated. The script, the claims and the source verification are human-authored and human-checked. Assets carry IPTC DigitalSourceType and C2PA Content Credentials.

Nothing asked, so nothing held — coding agents ship functional code nine times in ten and secure code one time in four, and the only AI Act date left this year is a label

The Last Secure Code Benchmark grades 450 tasks built from 150 real vulnerabilities with two hidden suites, one functional and one that replays the original proof-of-concept attack. Six frontier agents pass functionally 73.6% to 91.3% of the time and functionally-and-securely only 16.4% to 25.1%. The mechanism is not capability: for the top agent the security-only rate (25.3%) and the both-suites rate (25.1%) differ by two tenths of a point, so almost everything that survives the attack replay also works, and the specification simply never named the attack. The regulatory reflex is the AI Act, which does not reach code quality: the next AI Act date is 2 December 2026, Article 50(2) machine-readable marking of synthetic output, a provenance duty; Annex III human oversight moved to 2 December 2027 under Regulation (EU) 2026/1744; and the Cyber Resilience Act essential requirements that actually speak to attack surface apply 11 December 2027. Acceptance criteria are the only live control.

Post

Six frontier coding agents were given 450 real tasks. They got the code working between 74% and 91% of the time.

Both working and secure: 16% to 25%.

Four things I took from it:

The best agent, Claude Fable 5 in Claude Code, scored 91.1% functional and 25.1% functional-and-secure. The agent with the highest functional score of all, 91.3%, finished last on the combined measure at 16.4%. Passing more tests predicted nothing.
Security is the binding constraint, not capability. For the top agent the security pass rate (25.3%) and the both-suites rate (25.1%) differ by two tenths of a point. Nearly everything that survives the attack replay also works. The reverse is not remotely true.
The exposure does not come from autonomy. Averaged over the six agents, functional pass falls from about 93% when filling in a single function to about 80% when rebuilding a source tree, while functional-and-secure stays flat between 19% and 23%. It was already there at the smallest unit of work.
The best model solved only 12.0% of the 150 scenarios securely at all three granularities. For the other 88%, something broke somewhere.

What is in your definition of done that an attacker would recognise?

#AgenticEngineering #AIAct #SecureCoding #EURegulation #AppSec

AI disclosure: the narration voice, the cover art and the brand ident animation in this video are AI-generated. The script, the claims and the source checks are mine.

First comment

Sources.

The paper, abstract and metadata record: https://arxiv.org/abs/2610.14766

Per-model leaderboard: functional, secure, both, and the all-three-granularities column: https://last-secure-code-benchmark.github.io/

Benchmark code, task construction and the grading harness: https://github.com/alibaba/last-secure-code-benchmark

AI Act Article 50 transparency obligations, including the machine-readable marking limb: https://artificialintelligenceact.eu/article/50/

Cyber Resilience Act timeline, separating the September 2026 reporting duty from the December 2027 essential requirements: https://digital-strategy.ec.europa.eu/en/policies/cyber-resilience-act

AI disclosure: the narration voice, the cover art and the brand ident animation in this video are AI-generated. The script, the claims and the source checks are mine.