CLAIM-2026-10-10-01

Across five benchmarks and six frontier model backends, about 34.4% of trials judged correct violate the task's requirements.

Why it matters

If you quote a coding-benchmark pass rate in a review or a procurement note, roughly a third of the passes behind it are requirement violations the suite could not see.

Sources

• arxiv2026testjack