Your harness, your number — benchmark scores, agent scaffolds and who owes the evaluation

A refactoring benchmark published on 10 August 2026 shows the same model scoring 21.8% under one agent scaffold and 41.2% under another, on identical tasks and identical limits — so roughly nineteen points of what gets quoted as model capability belongs to the harness. The EU duty that speaks to evaluation quality, Article 55(1)(a), asks for protocols 'reflecting the state of the art' and binds the provider of the general-purpose model, and Annex XI Section 2 asks that provider for the metrics and the methodology on identifying limitations. Neither reaches the scaffold a deployer builds. The accuracy of the system anyone actually ships is therefore undocumented by construction unless the deployer documents it.

Post

Your coding agent scored 41.2%. Swap the scaffold and the same model scores 21.8%.

SWE-Bench ProMax was published on 10 August — a multilingual refactoring benchmark from a seven-institution team led out of Shanghai Jiao Tong, with Peking University, HKUST, Douyin Group, UCAS, NUS and Monash. 170 expert-curated instances across Python, Java, TypeScript, Go, C, C++ and Rust, averaging 11.4 modified files and 261.6 lines per task.

Best model: GPT-5.2, 41.2% resolve rate. On SWE-bench Verified, frontier agents now clear 75%.

But the number I keep rereading is the other one. GPT-5.2 scores 21.8% under the mini-swe-agent scaffold and 41.2% under OpenHands. Same model, same 170 tasks, same $10 cost cap. Nineteen points of "capability" that belong to the harness.

Two more findings. Gemini-3-Pro scores 0.0% on TypeScript while leading elsewhere — capability is per-language, not global. And GLM-5 and Qwen3.5 both reach 36.5% at $0.24 and $0.78 per instance, against GPT-5.2 at $3.60 and Claude Sonnet 4.6 at 38.8% for $4.77.

Background, and it is from February, not this week: OpenAI stopped evaluating on SWE-bench Verified. It audited 138 failed problems and found 59.4% had material test-design issues — 35.5% with tests too narrow to accept a correct fix, 18.8% checking behaviour the issue never specified. Every frontier model it tested could reproduce gold patches verbatim.

Now the European part. Article 55(1)(a) requires providers of general-purpose models with systemic risk to evaluate "in accordance with standardised protocols and tools reflecting the state of the art". Annex XI asks for the evaluation strategy, the metrics, and "the methodology on the identification of limitations". The AI Office began enforcing on 2 August.

Read the boundary. That duty lands on the model provider. The scaffold is yours. No provider evaluation, however state-of-the-art, describes the accuracy of the thing you deploy.

What I'd do Monday:

  1. Pin the scaffold as a versioned dependency. It is load-bearing, not config.
  2. Re-run your eval on a scaffold change alone. Keep the delta on file.
  3. Evaluate per language, not in aggregate. A 0.0% hides inside a good average.
  4. Write your limitations down before a customer writes them for you.
  5. Price the open models honestly. A few points can cost fifteen times more.

What is your agent's resolve rate on your stack, not the vendor's?

#AgenticAI #EUAIAct #AIEngineering #SoftwareEngineering #Berlin

AI disclosure: the narration voice, the cover art and the brand ident animation in this video are AI-generated. The script, the claims and the source checks are mine.

First comment

Sources, primary first —

SWE-Bench ProMax, arXiv 2608.09802, submitted 10 August 2026 — the 170 instances, the seven languages, the 41.2% best result and the 21.8% / 41.2% scaffold split. Affiliations taken from the paper's own front matter. https://arxiv.org/abs/2608.09802

The dataset itself, if you want to run it rather than read about it https://huggingface.co/datasets/swe-bench-promax/SWE-Bench-ProMax

OpenAI, 23 February 2026 — the audit of 138 failed SWE-bench Verified problems, the 59.4% figure, and the gold-patch memorisation finding. Note the date: this resurfaced in coverage this week, but the post is from February. https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/

AI Act Article 55 — paragraph 1(a), "standardised protocols and tools reflecting the state of the art" https://artificialintelligenceact.eu/article/55/

AI Act Annex XI — Section 2, point 1: evaluation strategies, criteria, metrics, and the methodology on the identification of limitations https://artificialintelligenceact.eu/annex/11/

European Commission, 2 August 2026 — the AI Office and national authorities begin enforcing https://ec.europa.eu/commission/presscorner/detail/en/ip_26_1714

One point cut for length: OpenAI recommended SWE-bench Pro as its replacement. Several outlets this week claim Pro has since been retracted over broken tasks. I could not verify that against a primary source, so it is not in the post.

Claims

Sources

Corrections

None.

AI disclosure

The video for this edition contains AI-generated media: the narration is a synthetic voice and the cover art is generated. The script, the claims and the source verification are human-authored and human-checked. Assets carry IPTC DigitalSourceType and C2PA Content Credentials.