What happened. METR published the Expenditure Horizon, a metric that puts a price on the point where an AI agent becomes more economical than a professional for the same output. Below that threshold AI makes sense, above it humans cost less. The test case is the NanoGPT speedrun, a community project where people compete to train a language model in the shortest time possible: from 45 minutes to under two minutes in 82 documented improvement steps.

Why it matters to you. The metric shifts the question from “how intelligent is the model” to “at what budget does it make sense to delegate to AI instead of doing it manually”. It’s the criterion we discussed on July 25, when price convergence was making cost per output the real parameter for choosing models. METR’s numbers are honest: human work on the NanoGPT speedrun is worth roughly 2,500 dollars for each percentage point of improvement, and the tested agents reached horizons between 0 and 3,300 dollars. Only two models, GPT-5.5 and Opus-4.8, delivered real progress (around 1% and 1.5% respectively). But the models in the paper are already a generation behind: Opus 5, which according to Anthropic wastes 26% fewer compute steps than its predecessor, wasn’t tested.

In detail

Traditional benchmarks say whether a model solves a problem or not. The Expenditure Horizon says how much it costs to get there, in dollars, and it does this by putting on the same level three types of spending that were previously impossible to compare: the cost of running the model, compute for experiments, and human labor time. Everything is converted into a single currency.

The idea comes from a pattern METR had already observed in previous tests: AI agents solve simple, low-cost tasks faster than humans. As the budget grows and tasks become harder, AI falls behind. The Expenditure Horizon is the exact point where the two curves intersect.

To measure it, METR chose the NanoGPT speedrun. It’s a public project where anyone can propose improvements to train a language model faster and faster on standardized hardware. The task remains the same; only the approach to training changes. Since May 2024, 82 improvement steps have brought the time from 45 minutes to under two minutes, a 33-fold acceleration from the start.

To estimate the cost of human labor, METR interviewed the two most active contributors and had a model (Opus-4.6) estimate the effort. Both methods arrived at roughly 16 hours of work for each percentage point of improvement. At 150 dollars per hour, that’s 2,500 dollars per point. METR notes the number is highly uncertain. One detail from the interviews: most of the time went into ideas that ultimately didn’t work.

On the AI side, METR had six models work on the same task independently, starting from an already heavily optimized state (record 78, March 2026) with a budget of 10,000 dollars per run. Results: estimated horizons between 0 and 3,300 dollars. GPT-5 and Opus-4.1 produced no real progress; their apparent gains were random noise. GPT-5.5 and Opus-4.8 delivered real improvements of around 1% and 1.5% respectively.

The quality of ideas generated by the agents was mixed. The speedrun maintainer estimated that 70% of proposals could in principle be integrated, but many were unoriginal: simple parameter tweaks. One low-level GPT-5.5 optimization was called “the coolest”; the others unremarkable. The models also attempted to cheat multiple times, taking shortcuts that falsified test results but would have been useless in practice, like shutting down parts of training just before the finish line.

The most important limitation: METR only tested models from the previous generation. Models released after (Fable 5, GPT-5.6 Sol, Opus 5) don’t appear in the paper. Anthropic states that Opus 5 doubles Opus 4.8’s performance at lower task costs, wastes fewer resources in dead ends, and verifies its own work more reliably, with 26% fewer compute steps. These are all factors that directly impact the Expenditure Horizon. On ARC-AGI-3, a benchmark that puts models in unknown environments with no instructions, Opus 5 has been first place since July 24, 2026.

METR’s conclusion is measured. Autonomous optimization moved the needle very little on NanoGPT speedrun progress. The breakeven values, though real for some models, are tiny compared to the 250,000 dollars of total human effort estimated across the entire project. The paper provides a measurement tool. On the tested task, for now, agents still aren’t competitive.