Coding Agent Task Gap: One in Four Tasks Fully Done

The coding agent task gap is now measurable on real app work: on Co-Design Bench, the strongest of ten models meets 69.9% of acceptance criteria on average but fully completes only 26.4% of tasks. Nearly three in four revisions still need a person to close one leftover requirement. Managers should staff review for that remainder, not for criterion averages that look like throughput.

Updated November 8, 2026

What is the coding agent task gap on Co-Design Bench?

Researchers at Alibaba and several Asian universities posted the paper on 26 October: 273 tasks from 37,463 deduplicated queries across 26,314 real coding sessions, split into 100 multimodal edits and 173 repository edits. Agents never see the 2,331 expert criteria. Requests can include an image, audio clip, video, or document, THE DECODER reported.

The Co-Design Bench paper states:

Co-Design Bench provides a testbed for studying the gap between partial requirement satisfaction and complete task success under agentic execution for real-user application revisions.

Grading uses a judge-then-refute protocol. One AI judge tests the finished app in a browser; a second re-checks every positive verdict for mock-ups or hardcoded results. Points go only to features that actually work.

Researchers favored cases where users had to rephrase a request multiple times or roll back changes, so the bench is mainly work that already troubled agents. The paper notes Opus 4.8 reaches 99.4% mean graded feature completion on 150 WebCompass-edit tasks under agentic execution.

How do full-task scores compare across models?

Tech Prolonged published full-solve rates for the leader, the next reported finishers, and last place. Only GPT-6 Astra is reported with both a criterion average and a complete-task rate.

Where does criterion completion drop by edit type?

All ten endpoints record lower criterion completion on repository edits spanning three or four change types than on repository edits spanning one or two. Layout, design, interaction, and content are the types that stack. THE DECODER notes that intertwined requirements of this kind are the norm in real projects.

On larger overhauls, smaller Qwen3.8-27B climbed to second, though the authors do not derive a general advantage from that placement.

Why do tokens and harnesses fail to finish tasks?

More tokens did not fix the miss. GLM-5.2 burned nearly six times as many tokens as GPT-6 Astra on smaller changes and still scored significantly worse. THE DECODER also cites a study that found barely any measurable benefit from elaborate harnesses, despite up to three times the tokens. Teams tracking coding-agent cost cuts should not treat a lower bill as a finished task.

The working environment moves the score. Qwen3.8-27B scored 75% or 56% on larger overhauls depending on the harness, and which of two models finished ahead also depended on that environment. The authors recommend reporting model and environment together, and testing that pair before a company picks an agent. The same check applies to assistants this site has covered, including GitHub Copilot Fusion.

What does this mean for review capacity?

As analysis of the published split, not a new result: one missed criterion fails the whole task, so 69.9% coverage can coexist with 26.4% complete delivery. Counting requirements touched will overstate how many agent runs are ready to merge.

For review staffing, the split implies:

• Separate full-task success from mean criteria passed.
• Expect heavier review when a repository edit mixes three or four change types.
• Test model and harness together before staffing a rollout.
• Do not read one-shot scores as evidence about mid-task corrections.

The test uses one fixed request per task, so mid-task corrections are not measured, and AI judges can be wrong. THE DECODER notes a METR finding that project-lead acceptance of AI-generated code ran roughly 24 percentage points below SWE-bench Verified scores. A 96-task multimodal subset is public on GitHub for a local rerun.