Two benchmarks released days apart measure the gap between test conditions and operational reality. SWE-Touch evaluates what happens when a human user modifies code while an AI agent is working on a repository. The classic SWE-bench assumed code remained unchanged from start to finish: in reality, developers keep committing while the agent works.

ExtractBench, which we discussed on August 3, puts agents in front of 4,869 pages of real business documents with complex layouts, instead of clean text. There too, frontier models truncate long records and coding agents cost too much relative to the value they deliver.

In both cases, the pattern is the same: traditional benchmarks measured the best an agent can do under ideal conditions. These two measure what happens when conditions aren’t ideal, and frontier models lose performance significantly.

For those deploying agents in production, the difference matters. SWE-bench tells you how much a model is worth when nobody touches anything. Real production has developers committing while the agent works, documents with broken tables, context that shifts. The delta between the score on the clean benchmark and the realistic one is the measure of your exposure when the agent leaves the demo.

In detail

The problem that SWE-Touch and ExtractBench address is structural. SWE-bench, the reference benchmark for coding agents, takes real issues from GitHub and measures whether an agent resolves the problem. But it freezes the repository at the moment the issue was opened: nobody touches the code while the agent works. In real life, a colleague pushes, a dependency updates, the file the agent is reading changes underneath it. SWE-Touch simulates exactly this: the human user modifies code during agent execution, and the benchmark measures how the agent reacts.

ExtractBench addresses a similar problem on the document side. Traditional data extraction benchmarks use documents with predictable structure. ExtractBench collects 67 types of real business documents, for 4,869 total pages, and measures four dimensions together: accuracy, completeness, traceability, and cost. Results show that vision-language models truncate records when they’re long, and coding agents navigating the document step by step spend too much for the value they deliver.

What unites the two benchmarks is the idea that evaluation should approach production conditions. Multi-dimensional measurement, not just accuracy but also cost and traceability, changes the judgment on models. An agent that scores 90% on SWE-bench can collapse when code changes mid-task, because its action plan was built on a state that no longer exists.

The limit of both is that they’re still simulations. SWE-Touch simulates human modifications according to predefined patterns, doesn’t capture the disruptive creativity of a real developer rewriting an entire file while the agent works on the old version. ExtractBench uses real documents but selects them, doesn’t cover all cases a real business document pipeline might encounter. For those who want a concrete sense of how much their agent holds up in production, the custom benchmark method remains the next step: take your own cases, make them messy, and measure.