A paper published on HF Daily Papers tackles a concrete retrieval problem: when an AI agent consumes the documents, what determines answer quality is how the full set of retrieved documents work together. Current evaluation systems score each document in isolation with metrics like nDCG and sum the results. They ignore document interactions: two texts can both be relevant yet redundant with each other, or they can contradict, or one can complete the other. Document-by-document evaluation misses this.

Why this matters to you

If you build RAG systems or search agents, the answer you get depends on the set of documents that ends up in your model’s context. The paper measures this: 12 rerankers tested on a benchmark spanning nine dimensions and roughly 28,000 rubrics, and the best barely reaches 45% coverage on document coordination dimensions. The method they propose, Rubric4Setwise, converts evaluation criteria into set selection signals and achieves better answers with fewer documents and fewer search rounds.

This connects to the thread we’ve been following: as we reported on July 20, three papers converged on the idea that in agentic systems architecture matters more than the model. Retrieval quality is one piece of that architecture.

In detail

The problem is real and anyone who’s built a RAG system in production recognizes it. You retrieve ten documents with semantic search, pass them to a reranker that orders them by individual relevance, and hand them to the model. Three documents say the same thing slightly differently, two contradict each other, and the piece of information you actually needed ranks sixth with a lower score.

The standard metric, nDCG, measures how well relevant documents rank at the top. It treats each document as an island. If two documents are both 90% relevant and say the same thing, nDCG rewards both. The agent receiving them has repetitive context and no new information.

The paper builds SetwiseEvalKit, a benchmark with three levels and nine evaluation dimensions, covering short and long format scenarios, with roughly 28,000 rubrics. The dimensions measure things nDCG ignores: redundancy (how much documents repeat), conflict (when they contradict), complementarity (when they complete each other), coverage (how much required information exists in the set).

Test results across 12 rerankers are clear. The best reaches at most 45% coverage on cross-document coordination dimensions. No method maintains best performance in both scenarios (short and long format). Cross-document dimensions are weak everywhere.

The operational proposal is called Rubric4Setwise. It’s a training-free method that takes evaluation rubric criteria and converts them into signals for selecting the document set, not individual documents. Result: better downstream answers with fewer documents and fewer search rounds. It’s the only method maintaining state-of-the-art results in both scenarios.

How much to trust this

The paper is based on the abstract and Hugging Face page: the full text wasn’t accessible at collection time. The numbers (12 rerankers, 45% coverage, 28,000 rubrics) come from the abstract and can’t be independently verified. A comment thread flags the absence of pointwise rerankers from tests, and the authors respond they classified them under “Adhoc Reranking”. The central question remains open: are evaluation dimensions (redundancy, conflict, complementarity) stable across domains, or do they need recalibration for each use case? The definition of a “good document set” shifts significantly between legal due diligence and code search.

For those wanting to dig into retrieval fundamentals, the original RAG paper remains the starting point.