DocOps, a deterministic evaluation framework for autonomous agents manipulating digital documents, is now available. It’s presented in a paper on arXiv (2607.19865) from the icip-cas team, with public code and project page.

The gap it fills is concrete. SWE-bench measures agents on code, and we know what works and what doesn’t. But operations on documents (PDFs, Word files, forms) lacked a systematic benchmark. If you’re considering entrusting an agent with contract compilation or editing a long report in your organization, you had no standard way to know whether the model you chose could handle it.

DocOps breaks down document operations into atomic dimensions and increasing complexity levels, drawing from real-world practices. Evaluation is deterministically verifiable, not left to an LLM judge that might skew results (a problem we’ve already seen in other benchmarks).

The paper’s results are honest about where we stand: even the most advanced setups show deep limits on tasks with many dependencies and long horizons. The three failure modes identified are most useful for builders: state loss on long tasks, shallow semantic verification, and destructive modification of structural metadata. These are exactly the flaws that ruin your day when an agent corrupts a file you can’t recover.

If you want to examine it closely, the repo is on GitHub (github.com/icip-cas/DocOps) with the project page at docopsbench.github.io.

In detail

The starting point is a real gap in agent evaluation. SWE-bench, which we discussed as a benchmark for code, showed how to measure agents on verifiable, realistic tasks. DocOps applies the same principle to a domain affecting almost every professional: documents.

The benchmark structure rests on a hierarchical taxonomy that decomposes document operations into atomic dimensions (individual elementary actions) and then recombines them into workflows of increasing complexity. The idea is to move from single steps (change this value in the form) to long processes (update a twenty-page report while maintaining coherence across sections, references, and metadata). Verification is deterministic: you check the final document state against an expected result, without relying on a model that votes.

The three failure modes the paper isolates deserve attention because they’re the kind of problem you discover in production, not in demos:

• Long-term state tracking collapse: the agent loses track of what it’s already done and how parts of the document connect. It happens when the task requires maintaining coherence between distant sections.
• Shallow semantic verification: the agent only surface-checks that the result looks right, without verifying that the meaning aligns with the document’s intent.
• Destructive editing of structural metadata: the agent modifies or deletes structural file information (styles, references, hidden fields) while working, corrupting the document in ways that don’t show immediately but emerge later.

These failure modes align with what we covered on July 20: orchestration and verification matter more than the single model (/en/radar/agent-failures-orchestration-over-model-three-papers). DocOps proves this on a new domain and does it with a verifiable method.

For those evaluating an agent for document tasks, the lesson is direct: test on long, coupled tasks, not just single operations. The paper suggests everything breaks there. And if you’re unsure where to start building your own benchmark, the course on how to evaluate without benchmarks is the right place.

Limitations: the paper is fresh (published July 22, 2026) and the abstract doesn’t report specific model scores. The code is public, but reading complete tables requires going to the repo or PDF. The upvote count on Hugging Face (2) suggests it hasn’t circulated widely yet, and should be read as a research contribution, not a settled verdict.