Scale AI has released HarnessOpt-Bench, a benchmark that measures frontier models’ ability to optimize the harness: the set of prompts, tools, control flow, memory, and orchestration code that surrounds a model inside an agentic system. The benchmark gives an LLM a target agent’s starting harness, a fixed evaluation budget, and then measures normalized gains on a test set that remains inaccessible during search.

Why it matters to you. As we reported on August 1 when Manifest shut down its router to focus on orchestration, the bottleneck in agentic systems has shifted from model selection to how you orchestrate it. HarnessOpt-Bench provides the first shared metric to measure exactly this. Results across 111 runs with 5 frontier models show two things: optimizer models distinguish themselves more than the harnesses that wrap them, and using the model’s native harness doesn’t guarantee better results.

For those building agents, empirical confirmation of an intuition that was already becoming operational. Now there’s a number to prove it. The piece on why agents fail already showed this qualitatively: almost never the model, almost always the steps.

In detail

Until now, benchmarks measured models. SWE-bench measures if an agent solves real GitHub issues, GAIA measures if an assistant answers questions requiring research and calculation, but none systematically measured the quality of scaffolding around the model: the prompts, tools, control flow, memory, orchestration code. HarnessOpt-Bench fills this gap.

The setup is designed to prevent overfitting. An optimizer model, paired with an encoding harness, receives a target agent’s starting harness, graded evaluation feedback, and a fixed evaluation budget. It modifies the harness, proposes a final candidate, and is judged on normalized gains against the starting harness on a test set it cannot see during search. A controlled execution environment enforces the boundary: it measures resources used by the target agent, prevents access to test data, and preserves candidate versions for audit.

The numbers tell two things. First: the choice of optimizer model matters more than the encoding harness through which it acts. Across 111 runs, the 5 frontier models tested separate clearly, while different harnesses produce less differentiated results. Second: the model’s native harness (the one it was trained with or that its producer provides by default) doesn’t win consistently. The advantage of using the same environment as the model’s producer is real on some tasks but disappears on others.

This means harness engineering work, what many are already doing by hand, has measurable impact and isn’t secondary to base model choice. If you change prompts, adjust control flow, compact memory, or redefine tools, you’re doing exactly what the benchmark measures.

Limitations are stated. Evaluations are expensive and stochastic, meaning measured gains can vary between runs. The 4 downstream tasks cover different work types but remain a narrow sample. The 5 frontier models are the protagonists of the moment, but the snapshot holds only until the next ones arrive. The paper itself concludes there’s ample room for improvement: gains vary widely by task and starting harness, and no model excels at everything.

For those building agents, the operational message is the natural follow-up to a trend this site has been tracking for weeks. The infrastructure in between: orchestration, memory, quality controls. These matter more than the base model. Now there’s a benchmark to prove it with numbers, which means you can measure whether an intervention on agent architecture actually delivers gains, instead of relying on impressions or a demo that works once out of three.