The Question

You ask a model to summarize a 40-page contract. The summary it returns contains only true statements. The problem is it skipped the automatic termination clause, the liability cap, and three key deadlines. Everything it wrote is correct. What it left out costs you a client.

This paper answers a straightforward question: how do you measure whether a generated response contains all the facts it should contain, beyond checking whether what it says is accurate?

Why It Matters to You

If you use AI to summarize, analyze, or synthesize documents, the risk you know best is hallucination: the model inventing a false fact. The evaluation tools in circulation today are built around this. They break text into atomic statements, search for each one in source material, verify if it holds up. This works well at catching liars.

It works poorly at catching omitters.

Factual completeness is the silent problem of daily AI work. A summary that misses the key balance sheet figure, an analysis that doesn’t mention the main risk, a report that leaves out three of seven agenda items: none of these contain an error. All contain a gap. And gaps are harder to catch than inventions, because there’s no sentence to flag as false. There’s an absence.

This Meta Research paper, published July 2026, builds a benchmark called GAMUT (Grounded Assessment of Multimodal Factuality) designed to measure this missing half. For anyone building evaluation systems or trying to understand how much to trust a generated summary, it’s a concrete shift in perspective.

What the Paper Says

The starting observation is that the facts composing a complete response rarely form a flat list. They have structure: some matter more than others, some form open sets where total coverage counts, some follow sequence (the steps of a process), some depend on others (a relationship between two facts that a yes/no checklist misses).

The authors propose a two-level meta-rubric. The first level is a structure organizing the content required for a complete response: hierarchy, importance weights, relationships between facts. The second level is a flat binary checklist (yes/no) generated mechanically from the first level’s structure. An LLM judge then evaluates each checklist item.

The idea is that complex structure serves thinking and defining what matters, but automated evaluation works better on simple binary questions. The meta-rubric bridges them: it captures structural richness and translates it into something a model can evaluate reliably.

To build the benchmark, the authors collected 1,813 questions based on real images from wearable devices across 10 different domains. Each question comes with a rubric with evidence, verified by expert annotators. The framework is modality-independent, so they released a text-only variant too.

They tested 14 models spanning frontier and open-weight options. The best result came from Gemini 3.1 Pro at 58.7%. The paper describes the benchmark as genuinely difficult, highly discriminative (it separates good models from weaker ones well) and robust to judge selection (changing which model does the evaluating doesn’t upend the rankings).

The number to remember is that 58.7%. The best model, on this benchmark, covers just over half the facts a complete response should contain. It says two things: the benchmark is hard, and completeness is an open problem.

How Much to Trust This

The paper is honest about its approach’s limitations, and I’ll report some here because they matter for readers.

First: building the rubrics requires expert human annotation. It’s not an automatic process. For 1,813 questions you need qualified human work, which means scaling this approach to new domains costs time and people. The paper doesn’t hide this cost.

Second: the benchmark is anchored to images from wearable devices. A text-only variant exists, but primary validation is on multimodal material. If your use case is summarizing contracts or financial reports, whether these results transfer to your domain isn’t guaranteed by the paper. It needs verification.

Third: the 58.7% as a top score should be read carefully. Models are evaluated on open-ended questions where completeness means enumerating sets of facts, not choosing among options. It measures how well a model covers alone a space of information defined by a rubric. A 58% complete summary might still suffice if the omitted facts matter least. The benchmark measures raw coverage, not coverage weighted by usefulness.

Fourth: I’m working from the abstract and paper page, not the full text. Methodological details on meta-rubric construction, annotation quality, and variance between judges are in the complete paper.

What to Do With This

For anyone using AI to synthesize and analyze, the practical implication is direct: stop evaluating only for precision. When you check a generated summary, don’t just hunt for false sentences. Hunt for gaps. The playbook Long Documents Without Silent Omissions works exactly on this: getting a model to read a long document in blocks with anchored citations and a coverage index, to spot what got skipped. The paper gives academic framing to a problem anyone using AI encounters daily.

For anyone building evaluation systems, the two-level meta-rubric is an idea applicable even without the benchmark. You can define the structure of content a complete response should have (level one), translate it to a binary checklist (level two), and have a model evaluate each item. It’s a pattern you can rebuild with your own data and rubrics, without needing GAMUT.

What not to conclude: the 58.7% doesn’t mean models are useless for synthesis tasks. It means that if you ask a model to cover alone the full space of relevant facts on an open topic, it misses about half. Give it structure (an outline, a list of points to cover, a rubric) and outcomes change, because the model no longer decides what’s complete. You do.

The most likely misreading is using this number as a reason not to use AI for synthesis work. The number says something else: that you need a completeness check, and that check is doable.

Going Deeper

• The playbook Long Documents Without Silent Omissions is the operational counterpart: how to notice what a model misses when reading a long document.
• The course lesson Evaluating Without Benchmarks and the paper Evaluating an Agent Without Benchmarks build the framework for thinking about evaluation on your cases, not on averages.
• The paper LLM as Judge explains how it works and where one model’s scoring of another goes wrong, the foundation GAMUT rests on for its scoring phase.