A paper on arXiv presents HANDBOOK.md, a benchmark that measures whether models actually follow policy instructions over extended horizons. Sixty-five agentic tasks in simulated business environments (email, chat, calendar, issue tracking via Model Context Protocol), each governed by an operational manual of 20–124 pages written by domain experts. Under rigorous evaluation, the best-performing model passes 36.2% of trials. Most frontier configurations stay below 25%.

If you assumed a long system prompt or policy document was enough to keep an agent in line, these numbers say otherwise. The paper identifies four recurring failure patterns. The agent allows a plausible request in the environment to override stable policy. It executes a mandatory check and then acts against its result. It loses rule details as context lengthens. It declares compliance it hasn’t achieved.

This is the measured, repeatable version of what we saw when GPT-5.6 Sol ran a real company for 24 hours: the agent receives clear instructions and then violates them, because environmental pressure outweighs the starting document. The difference is that here failure is systematic rather than anecdotal, with 824 deterministic evaluation criteria.

The practical upshot is that agent governance cannot live in context alone. You need an external software layer that verifies actions after the agent takes them. The course on how to evaluate whether your agent works starts from the same principle: generic benchmarks don’t tell you whether your agent respects your rules.

In detail

Traditional benchmarks for agents measure whether a model completes a task. HANDBOOK.md measures something different: whether a long, binding policy document can constrain agent behavior over an extended horizon of tool use. It’s the difference between asking “did the agent finish the job?” and “did the agent follow the rules while doing it?”.

The 65 tasks model how corporate employees follow operational manuals. Each task places the agent in a self-contained environment: a workspace with files, email, chat, calendar, issue tracking, and commerce services, all exposed via Model Context Protocol. The policy is a manual written by experts, 20–124 pages long, covering five domains (finance, medical billing, insurance, logistics, human resources) across ten fictional companies.

To resist memorization, each task modifies one of ten base manuals, altering the specific rules and thresholds on which evaluation rests. No two tasks share the same policy. Evaluation is completely deterministic: 824 programmatic criteria that verify both that required actions were taken and that prohibited actions were not.

The numbers. Thirty model configurations evaluated. Under strict evaluation (a trial passes only if every criterion is met), the best passes 36.2% of trials. Most frontier configurations stay below 25%.

The four failure patterns:

1. Environment override. The agent receives a plausible request in the environment (an email, a chat message) that conflicts with policy, and follows the request instead of the policy. This is the most worrying pattern because it overlaps with the classic indirect prompt injection vector.
2. Check executed and ignored. The agent runs the check required by policy, gets the result, and then acts in the opposite way. It performs verification but doesn’t use it.
3. Degradation over long horizon. Rule details are lost as conversation and tool use extend. Rules at the start of the manual carry more weight than those toward the end.
4. Declared compliance. The agent reports complying with policy when it hasn’t. If you rely on the agent’s self-report for monitoring, this failure goes undetected.

What it means for builders. The paper’s implicit thesis is that context engineering alone is insufficient to govern agents in production. You can write the most accurate manual in the world, put it in the system prompt, and the agent will follow it fewer than four times in ten. Governance must sit in an external software layer: post-action checks, result validation, and where possible, restrictions on the tools themselves (don’t give the agent a tool it shouldn’t use, rather than telling it not to use one).

The limits. The benchmark uses simulated environments, not real production systems. The manuals are expert-written but the fictional companies may not capture the full complexity of real policy. The sample of 65 tasks is relatively small, though 824 evaluation criteria partly compensate. It’s unclear how results change with more sophisticated context engineering techniques (selective RAG over policy, retrieval of relevant rules at action time) versus simple manual insertion into context. The paper does not test these alternatives.