A group of researchers published WorldCupArena, a benchmark that evaluates language models and deep-research agents on soccer match prediction. Before each match, the system receives an information package or searches on its own, then predicts result, score, key players, and statistics. After the match, predictions are compared against the actual outcome. The first evaluation covers 104 matches from the 2026 World Cup across 13 systems.

Why it matters to you. Traditional benchmarks measure what a model knows at training time. WorldCupArena measures what an agent can discover and combine when data changes continuously. It’s the same problem you face when asking an assistant to prepare an offer with today’s news, or analyze a moving market. The ability to search for fresh information, filter it, and take a position before the answer is available is the skill that separates a useful agent from a model that repeats the past.

The results are honest: compared to bookmaker odds and human fans, the best system shows small gains on result and exact score. Net gain arrives on the Scoreline metric, which gives partial credit when the predicted score is close to the actual one. Models with the same accuracy on result differ on detailed predictions, where reasoning counts more than memory.

If you want to try: code, prompts, and evaluation scripts are open source on the project’s GitHub repo.

In detail

What came before.

LLM benchmarks have a structural problem: they measure knowledge frozen at training time. MMLU, HumanEval, SWE-bench all test things the model may have already “read” in training data. Even GAIA, which we described as the most honest thermometer for general assistants (the paper is here), uses questions with answers already known at test time.

WorldCupArena flips the approach: it picks events that haven’t happened yet. The model can’t have the answer in its weights, because the answer doesn’t exist yet. It must search for it, combine it, and take a position before the match is played.

How it works, in simple terms.

Before each match, the system receives one of two things: a standardized evidence package (same data for all models, to compare them fairly) or access to free web search. The model must then predict the result (win, draw, loss), exact score, players and key events, match statistics, and tournament outcome.

After the final whistle, each prediction is compared against the recorded result. The score isn’t binary: the Scoreline metric assigns partial credit when the predicted score is close but not exact, and this is where models differ most.

What the numbers say.

104 matches, 13 systems tested. The most interesting data is the comparison with two external baselines: bookmaker odds and human fan predictions. The best system beats these baselines slightly on result and exact score. Ground gain comes on the Scoreline metric and detailed predictions.

What this means: bookmakers are already very good at predicting who wins, because they’ve been doing it for decades with refined statistical models. An AI agent’s advantage emerges in finer analyses (who scores, how the match unfolds, which events are likely). Models with the same rough accuracy on result separate when you look at the details.

Limitations.

The sample is 104 matches: solid but not huge, and soccer is notoriously unpredictable. Small gains on result predictions might reflect the task’s intrinsic difficulty more than model limits. The benchmark is designed to be reused though: new schedules are added as competitions proceed, and future models are evaluated without old results already being known.

The paper was published on Hugging Face on July 20, 2026, with code, prompts, and evaluation scripts open source on the project’s GitHub repo.