The sector is not the test — agents clear a quarter of real professional work, and Annex III classifies by neither healthcare nor finance

DAYJOB scores agents on 130 tasks written by working healthcare and finance professionals, and the ceiling is under 25 percent. The regulatory reflex - two high-risk sectors - is wrong twice over: the AI Act does not ask how well a system works when it classifies it, and Annex III point 5 names use cases about natural persons, not industries. Payer eligibility and consumer credit scoring are in; a portfolio memo, a corporate credit file and a clinical research summary are not. The Article 6(3) carve-out that most agent products actually sit on dies on profiling, and claiming it costs a documented assessment before placing on the market plus registration under Article 49(2) - a duty keyed to when you ship, not to the Annex III date of 2 December 2027.

Post

A two-sentence email and twenty-five attachments. That is what a real professional request looks like. It is now a benchmark.

On Thursday Surge AI published DAYJOB: 130 tasks written by working healthcare and finance professionals, scored on whether the work is usable by a colleague.

Flat:

  1. The strongest configuration cleared 24.7% in healthcare and 23.9% in finance. The median of 30 configurations from 13 developers cleared 0.6% and 2.5%. In healthcare the leader is more than double the runner-up.

  2. The briefs are short on purpose: 81 words on average in finance, 49 in healthcare. Rubrics carry a median of 57.5 binary criteria in finance, 47.5 in healthcare, graded by a model acting as an agentic judge inside the container. Scope the work, then be complete, then survive a grader that is itself an agent.

  3. Before you read 24% as "nobody is shipping this": the AI Act does not ask how well a system works when it decides what it is. And if your instinct is that healthcare and finance are high-risk sectors — Annex III names use cases, not industries. 5(b) is creditworthiness of natural persons, fraud detection carved out; 5(c) pricing for life and health insurance; 5(a) public authorities deciding eligibility for benefits, healthcare included. A portfolio memo, corporate credit, a clinical research summary: not listed. Payer eligibility and consumer credit scoring: listed.

  4. The carve-out people reach for is real, and narrower than it reads. Article 6(3) lets you out for a narrow procedural task, improving completed human work, deviation detection that does not replace review, or a preparatory task — never where the system profiles natural persons. Article 6(4) then makes you document that assessment before the system is placed on the market and register under Article 49(2). Annex III obligations move to 2 December 2027 under Regulation (EU) 2026/1744; the documentation duty is keyed to when you ship.

My Monday: write your system's intended purpose as one sentence, then read Annex III point 5 against it with "natural persons" underlined. If you land outside, write the Article 6(3) assessment now and date it — in Germany the Bundesnetzagentur would ask for it. And make your acceptance rubric binary criteria, not a summary score.

A sector is not a classification. A sentence you have not written is not a carve-out.

What is your system's intended purpose, in one sentence?

#AgenticAI #EUAIAct #AIGovernance #HighRiskAI #AIEngineering

First comment

Sources, the paper first.

DAYJOB on arXiv — 130 tasks, the 24.7% and 23.9% ceilings, and the median configuration at 0.6% and 2.5% across 30 configurations from 13 developers: https://arxiv.org/abs/2610.01306

Surge AI's own write-up — the 81-word and 49-word prompt averages against 337 for a comparable benchmark, the input-file counts, and the three-layer expert review: https://surgehq.ai/blog/dayjob

The finance leaderboard — 80 assignments across corporate finance, banking, credit, investing and real assets, and the "usable by someone else" scoring standard: https://surgehq.ai/benchmarks/dayjob-finance

The healthcare leaderboard — clinical, pharmacy, operations, payer and compliance, and the full 32-row board: https://surgehq.ai/benchmarks/dayjob-healthcare

The evaluation harness — this is where you see that grading is an agentic LLM judge running inside the task container, Apache-2.0: https://github.com/surge-ai/dayjob

AI Act Article 6 — read 6(3) for the four derogation limbs and the profiling sentence, then 6(4) for the documentation and registration duties that come with claiming one: https://artificialintelligenceact.eu/article/6/

Annex III — point 5 is the one that matters here, and "natural persons" is the operative phrase in 5(b) and 5(c): https://artificialintelligenceact.eu/annex/3/

Article 49(2) — the registration you owe even when you have concluded your system is not high-risk: https://artificialintelligenceact.eu/article/49/

Regulation (EU) 2026/1744, the Digital Omnibus on AI, in force 27 July 2026 — the instrument that moved the Annex III date: https://eur-lex.europa.eu/eli/reg/2026/1744/oj

A clean summary of the new dates: Article 50 since 2 August 2026, marking from 2 December 2026, Annex III to 2 December 2027, Annex I to 2 August 2028: https://www.hunton.com/privacy-and-cybersecurity-law-blog/eu-digital-omnibus-on-ai-enters-into-force

The Bundesnetzagentur's own announcement of its role under the AI Regulation, 29 July 2026: https://www.bundesnetzagentur.de/SharedDocs/Pressemitteilungen/DE/2026/20260729_KI_VO.html

KI-MIG, the German implementing act, in force since 29 July 2026: https://www.gesetze-im-internet.de/ki-mig/

One point I cut for length: the paper reports 130 tasks split 50 healthcare and 80 finance, but the live healthcare leaderboard now lists 65 assignments. The set has grown since submission, which is the honest reading, and it is a reminder that a leaderboard and the paper describing it are two different documents. I quoted the paper's figures throughout and the scores off the boards.

Claims

Sources

Corrections

None.

AI disclosure

The video for this edition contains AI-generated media: the narration is a synthetic voice, the cover art and any background imagery are generated, and the brand ident animation is generated. The script, the claims and the source verification are human-authored and human-checked. Assets carry IPTC DigitalSourceType and C2PA Content Credentials.