FinanceComplexQA is an open-ended benchmark for evaluating how well AI agents and RAG systems perform on industrial financial documents. Researchers generated 2,000 professional documents using a system called Finance-LaTeX SKILL, which produces complex layouts (nested tables, footnotes, cross-references) from expert knowledge. Across those documents are 2,026 deep research tasks with 6,000 question-answer pairs.

The angle for anyone using AI in their field is the variance between agents. The paper confirms what anyone who’s tried to have an assistant read balance sheets or prospectuses already knows: some agents fail on numeric calculations, others lose track in multi-hop reasoning (where you need to connect information scattered across different pages), still others summarize well but skip context analysis. The benchmark measures this systematically, with an Agent-as-a-Judge that evaluates open-ended responses on multiple metrics instead of seeking exact match with a reference answer.

Why it matters to you: if you’re thinking about having AI read documents in your field, this work tells you where to look to understand whether the agent you’ve chosen will hold up. The documents have the same characteristics that break extraction in real life.

As we covered on July 23 with DocOps, measuring agents on documents is the gap that SWE-bench (built for code) didn’t cover. FinanceComplexQA goes vertical: same principle, domain-specific.

In detail

Context: benchmarks for agents have multiplied over the past year, but most cover code (SWE-bench and derivatives) or generalist tasks (GAIA). Financial documents are a particular domain because they combine three difficulties that rarely coexist: numeric calculations on dense tables, multi-hop reasoning across sections linked by cross-references, and context analysis requiring knowledge of the industrial scenario framing what you’re reading.

The interesting methodological choice is synthetic document generation. Rather than collecting real financial reports (with licensing, privacy, and insufficient variety issues), researchers built an agent, Finance-LaTeX SKILL, that synthesizes documents with complex layouts from expert knowledge. The advantage is control: they can generate cases with any combination of difficulties and know what the correct expected answer is because they built it. The disadvantage is that documents are synthetic, and a model that saw millions of real financial statements during pre-training might behave differently on invented layouts.

Evaluation uses an Agent-as-a-Judge: a model evaluates open-ended responses from tested agents on multiple metrics, rather than comparing them to a reference answer. It’s the same pattern we’ve seen in other recent benchmarks, and it carries the same problem we discussed in the paper on LLMs as judges: the judge has its own biases. Researchers claim reference answers are “relatively stable and permanent,” but open evaluation remains the methodological weak point.

Key numbers: 2,000 documents, 2,026 tasks, 6,000 question-answer pairs, bilingual support (Chinese and English), six mainstream scenarios and seven task types. Scale is in the mid-range for domain-specific benchmarks.

What remains open: the paper is available as a Hugging Face page, but the full text wasn’t retrieved for this entry. Individual agent performance metrics aren’t reported in the abstract, so we can’t say who wins. Documents are synthetic, so results might not transfer directly to real financial statements professionals encounter at work. And as always with benchmarks, the score tells you what happened on that set, not what your agent will do on your documents: for that you need a test bed built on your cases, as explained in lessons on evaluating without benchmarks and building your own test bench.