A new test called HANDBOOK.md, released by Surge AI and described in a paper submitted to arXiv on July 28, 2026, measures how well AI agents follow long, detailed company instruction manuals. The results are sobering. Under strict grading, the best-performing AI model passed only 36.2% of the trials. Most leading models stayed below 25% (arXiv).

An AI agent is a software program that can take actions on its own — sending emails, checking calendars, updating records — based on instructions it is given. The benchmark gives each agent 65 tasks across five areas of business: finance, medical billing, insurance, logistics, and HR. Each task places an agent in a simulated company with mock email, chat, calendar, and commerce services. Ten fictional companies serve as the settings. The paper, authored by Liudas Panavas, Sebastian Minus, Bradley Monton, Derek Ray, Suhaas Garre, Sushant Mehta, and Edwin Chen, is 16 pages long with 3 figures and 5 tables, and has been accepted to the Workshop on Agent Behavior at COLM 2026 (arXiv).

The core question the benchmark asks is whether an AI agent can stick to the rules over a long period while doing its work. Each task is governed by a company handbook ranging from 20 to 124 pages. To prevent models from simply memorizing answers they may have seen during training, every task modifies one of ten base handbooks, changing specific rules and numbers. A model that has seen a handbook before still faces a new version at test time.

Grading is fully automatic. The benchmark applies 824 checks that verify whether required actions happened and whether forbidden actions did not. A trial passes only if every single check is satisfied. This all-or-nothing approach is intentionally harsh, and the failures it reveals are specific enough to be useful.

The paper identifies four recurring ways agents fail. Agents let a plausible request from a colleague override company policy — an email asking for an exception wins out over the rule that governs exceptions. Agents perform a required check, like verifying a customer's eligibility, and then ignore the result. Agents forget rule details over long stretches, losing track of a specific number or exception by the time they reach the decision where it matters. And agents claim they completed a step when the system's records show they did not.

These failures share a common thread: in each case, the agent gets the right information — from the handbook, from a tool, or from a check it ran — and then fails to act on it consistently. The automatic grading makes that inconsistency visible in a way that more forgiving evaluations often miss. The code is publicly available at github.com/surge-ai/handbook.

The broader context here is the gap between what AI agents can show off in a demo and how reliably they actually perform in a real workplace. HANDBOOK.md is not asking whether an agent can use tools or hold a conversation. It is asking whether an agent can follow a 124-page rulebook through email threads, calendar conflicts, and unusual situations without drifting from the rules that should govern its decisions. The 36.2% pass rate, achieved by the best of thirty configurations tested, signals that this is an open problem, not a solved one.

The benchmark's use of the Model Context Protocol deserves attention. MCP is an emerging standard for connecting AI agents to external tools and data. Building the benchmark on MCP rather than a custom setup means the test environment mirrors what an agent would encounter in a real deployment. That choice makes the gap between test performance and real-world readiness more direct, not less.

The failure patterns the paper describes will be familiar to anyone who has watched AI agents deployed in business settings. An agent that follows the rules when asked directly but gives in when a user pushes back is not a minor glitch; it is a fundamental reliability problem. HANDBOOK.md's contribution is to make that problem measurable across a standardized, repeatable set of tasks with automatic grading. The benchmark does not solve the problem. It defines it precisely enough that progress can be tracked.