What happened.

Two papers published in July propose benchmarks that measure coding agents on what they actually do in 2026: take a vague intention and build a working project on top of it. ICAE-Bench (480 tasks, 12 languages) simulates a user handing an agent an incomplete product request, evaluating planning, requirement clarification, debugging, and repository-level integration. Tencent’s WorkBuddy Bench constructs reverse-engineered tasks from real commits and pull requests, across four domains: code, web, office, security.

Why this matters to you.

If you use Claude Code or Codex, you know the hard part isn’t asking it to write a function anymore. The hard part is explaining what you want when you don’t fully know yourself yet. Traditional benchmarks like SWE-bench, as we covered in the guide to the paper that defined the field, give the agent a GitHub issue with precise instructions and measure whether it resolves it. ICAE-Bench and WorkBuddy start from a different premise: the input is fuzzy, the agent must ask questions, make design decisions, handle hidden constraints.

ICAE-Bench’s results make it clear: agents reproduce visible behavior but struggle with hidden constraints, edge cases, and integration over long horizons. This is exactly the point where your work on context-setting and spot-checks makes the difference between a working prototype and software that holds up.

In detail

The paradigm shift is sharp. SWE-bench, the benchmark that has defined how we measure coding agents so far, starts from GitHub issues with already-precise descriptions: there’s a bug, there’s a requested feature, the agent must produce a patch that passes tests. It’s a model that matches the work of a programmer receiving a well-written ticket.

Vibe-coding, the term that spread around using agents to build software from informal descriptions, has shifted the center of gravity. People using Claude Code or Codex no longer write complete specifications: they throw out an idea, the agent asks questions, it iterates. The benchmarks needed to adapt.

ICAE-Bench tackles the problem with an interesting architecture. Each task derives from a real open-source repository with verified executable behavior. The ambiguity isn’t invented: it comes from the difference between what a user would ask in words and what the code actually needs to do. An automated User Agent simulates the human counterpart, revealing hidden constraints as the agent asks questions, without inventing new requirements or leaking the solution. Evaluation combines standardized black-box tests with multi-dimensional diagnostics: functional correctness, semantic and API similarity, structural fidelity, design quality, interaction quality.

Results across six models and two agentic frameworks show a recurring pattern. Agents succeed at reproducing the application’s visible behavior but fail on hidden constraints, edge cases, and long-horizon integration. The demo works, the details don’t.

Tencent’s WorkBuddy Bench takes a different route toward the same goal. Instead of adapting text from public issues, each task is reconstructed backward from a real commit, pull request, or corporate scenario, then reformulated to resist contamination (the risk that the model has already seen the solution in training data). It covers four domains: code, web, office, security. The focus on contamination is relevant: when benchmarks rely on public repositories, larger models have a spurious advantage because they’ve memorized code during training.

What changes for those using these tools? Two practical things. First, when you read a coding agent score, check what it measures: a high number on tasks with precise specs tells you little about how the agent will handle your vague Friday afternoon request. Second, the weak points that ICAE-Bench highlights (hidden constraints, edge cases, long integration) are exactly where your work on clarification and spot-check verification makes the difference between delivering a prototype and delivering usable software.

The limitations of both benchmarks should be stated. ICAE-Bench simulates the user with a model, which introduces typical LLM-as-a-judge variability: the User Agent could be more or less collaborative than a real human. WorkBuddy is built by a Tencent team and complete methodology documentation is in the paper, but the cross-model leaderboard isn’t yet publicly available in independently verifiable form.