ExtractBench is a benchmark for schema-guided data extraction: given a document and a schema you define, the agent must return the correct values with an indication of where it found them. It covers 4,869 pages across 370 business documents, 8 business domains and 67 document types. Published on arXiv on July 31, 2026, it’s the first benchmark to measure together value accuracy, record completeness, source traceability and cost.

If you use agents to extract data from documents (invoices, contracts, price lists, forms), this changes the question you ask before choosing them. The paper shows a clear pattern: commercial VLMs perform well on short documents but truncate record lists when the document gets longer. Coding agents maintain accuracy, but at a much higher cost. LlamaExtract Agentic Plus wins on all three metrics with accuracy comparable to coding agents at a fraction of the cost.

This is the thread we followed with DocOps: SWE-bench measures code, but documents remained uncovered. ExtractBench fills the gap with a concrete test bed. The paper’s point is the method: accuracy evaluated as record completeness (not as individual values correct), and cost measured alongside quality.

Dataset and evaluation code are public on HuggingFace and GitHub: you can test your agent on the same documents and compare.

In detail

What came before.

Agent benchmarks focused on code (SWE-bench) or general tasks (GAIA). For documents, there was no test bed that measured schema-guided extraction at real scale. DocOps had identified the gap; FinanceComplexQA had partially filled it for the financial domain. ExtractBench addresses it across 67 document types and 8 business domains, with 4,869 pages total.

How evaluation works.

Schema-guided extraction works like this: you define the structure you want to get (for example a JSON with fields “vendor”, “date”, “line items”, “total”), and the agent reads the document and fills in the fields. ExtractBench measures three things:

• Value F1 order-insensitive: how many values it extracted correctly, regardless of the order they appear. Penalizes both made-up values and missing ones.
• Grounding word-level F1: for each extracted value, the agent must indicate the exact words in the document it came from. Verifies provenance at word level.
• Grounding page-level F1: same thing at page level. For long documents, knowing “page 14” is already operational information.

Cost is measured in dollars per document, reported alongside quality metrics.

How ground truth is built.

The challenge of every benchmark is having the right answers to compare the agent against. ExtractBench uses three strategies depending on document type:

• Real documents: two independent systems extract the same data. If they agree, the value is considered correct.
• Synthetic lists: values are known because artificially generated, so comparison is exact.
• Forms: human verification, because irregular layouts make system agreement insufficient.

What the numbers say.

Commercial VLMs do well on short documents, but truncate record lists on long documents. It’s a silent failure: the agent doesn’t get the values it extracts wrong, it extracts fewer than it should. If you process a 200-line price list and the agent returns 120 lines, you get output that looks correct but is incomplete.

Coding agents maintain accuracy on long documents, but spend much more. The trade-off between accuracy and cost is the practical heart of the paper.

LlamaExtract Agentic Plus, made by LlamaIndex, ranks first on all three metrics, with a cost comparable to VLMs and accuracy comparable to coding agents. It’s the point on the curve where you spend little and get a lot, at least on this benchmark.

Limitations.

The paper came out days ago and the dataset is new. The 370 documents cover 8 domains and 67 types, but representativeness for your specific domain needs verification. Ground truth for real documents is based on agreement between two systems: if both get it wrong the same way, the “correct” value is wrong. The GitHub repo is run-llama/ExtractBench, LlamaIndex’s organization: the fact that its own tool wins the benchmark should be read with this in mind, even though code and dataset are open and replicable.