OpenAI publishes results on ARC-AGI-3: two API settings, reasoning enable and context compaction, triple GPT-5.6 scores. How many additional tokens do those settings cost, and how much does latency increase? The post doesn’t say.

JuliaHub published an independent evaluation this week that includes the cost. They put three variants of GPT-5.6 and Claude Fable 5 to work solving five physics simulation problems in a controlled environment, with sealed ground truth. The numbers: Claude Fable 5 scores 0.889 weighted score at $9.60 per trial, GPT-5.6-terra does 0.786 at $1.25. Fable 5 is more accurate, but costs almost eight times as much per attempt.

As we reported on July 25, the cost of a usable response is becoming the real decision criterion. OpenAI’s post confirms this despite itself: if the two parameters were free, they would highlight it.

The problem is methodological. A benchmark that counts only correct answers, without weighing the cost to produce them, tells half the story. If tripling the score requires burning ten times the tokens, the operational decision changes. For anyone evaluating models on a real case, the Compare Two Models in a Quarter Hour playbook remains the most honest method: same task, same test cases, quality-cost-latency table on your data.

In detail

What ARC-AGI-3 measures.

ARC-AGI is François Chollet’s benchmark of grid-based visual puzzles: each task is unique, requires fluid reasoning and memorization resistance. ARC-AGI-3 is the latest version. It’s the kind of test where a model can’t get by on memorized patterns: it must generalize from single examples.

The two parameters.

OpenAI’s post discusses two API settings. The first, reasoning enable, keeps the reasoning chain active between turns instead of starting over from scratch. The second, context compaction, compacts the conversation context to fit more material in the window. Both trade tokens (and time) for accuracy. The post announces a tripling of the score and mentions “efficiency,” but doesn’t publish token consumption or measured latency.

JuliaHub’s numbers.

JuliaHub tested the models on a different domain (physics simulation, not visual puzzles), but with a methodology that includes cost. Locked configuration: reasoning effort xhigh, 1M context window, 128k token budget. Sealed problems with ground truth hidden from the model.

Results:

• Claude Fable 5: 0.889 weighted score, $9.60 per trial, 16.1 minutes
• GPT-5.6-sol: 0.814, $1.74, 13.4 minutes
• GPT-5.6-terra: 0.786, $1.25, 12.6 minutes
• GPT-5.6-luna: 0.727, $3.26, 25.0 minutes

Fable 5 wins on accuracy, but GPT-5.6-terra costs 7.7 times less per trial. In a workflow where you need to iterate many times, or where “good enough” suffices, the more economical model wins on total cost.

The blind spot of agentic evaluation.

JuliaHub raises a deeper issue: in physics simulation, a model can compile and run cleanly while encoding impossible physics. Agents make it worse because they write their own tests. The question “does this match the real world?” is harder to settle than “does this pass the tests?” It applies to anyone building agents that work in domains where verification isn’t straightforward.

What it means for readers.

If you’re choosing a model for a repetitive task (extracting data from documents, classifying tickets, generating drafts), the cost per usable output matters more than a score on an academic benchmark. A model scoring 10 points higher but costing five times as much isn’t always the right choice: it depends on how many times you run it and what one more error is worth. OpenAI’s two parameters exist in the API: whoever activates them should track token consumption before and after on their own workload, not on ARC-AGI-3.

Limitations.

JuliaHub data covers five problems in a single domain. OpenAI’s numbers are on ARC-AGI-3, a different benchmark. The two datasets are not directly comparable. What they share is the methodological lesson: include cost in your evaluation. A tripled score without the price in tokens remains a marketing claim, not an operational decision.