Microsoft's Thinkingbox grades agents on the database: 66.5% once, 47.5% every time

Microsoft has released Thinkingbox, a sandbox that grades AI agents on what they changed in a backend database rather than on what they said they did, together with Thinkingbox-bench, a 507-task benchmark of stateful business workflows built in partnership with Toloka. The headline result is a gap rather than a score: the strongest model in the paper, Claude Opus 5, completes a task on its first attempt 66.50% of the time, but succeeds on all 20 attempts for only 47.53% of tasks.

The paper, One Success Isn't Reliability, is by Zhuochun Li (University of Pittsburgh), Youngmin Ko (Northwestern) and Ali Keramati (UC Irvine) — all three working during Microsoft internships — with nine Microsoft coauthors. Its argument is that agent evaluations which score a final response or a valid tool call are measuring the wrong object. As Microsoft engineer Liang-Chun Tsai puts it in the company's write-up, an agent asked to add a quiet-room preference to a hotel booking can collect the reference, check the reservation and confirm the change while the booking's special_requests field stays empty: "The agent confirmed work it never did. A grader looking only at the final response might pass the run. You discover the failure at check-in."

Figure 1: the sandbox loop, and the discovery–reliability gap across 507 tasks and 20 attempts each. Credit: Li et al., arXiv:2608.19741.

#How the benchmark works

Each task is a runnable world: an initial backend state, a user goal, MCP-compatible domain tools, a simulated user who releases facts only when the agent asks for them, and hidden executable checks. Every attempt gets a fresh, isolated tool session, so no run inherits another's side effects. The verdict is conjunctive — a task passes only when every check passes — and the checks reject wrong, missing or extra effects, which is how the benchmark catches an agent that updates the right record and one it should have left alone. Thirty tasks additionally apply binary rubrics to the final message, covering required disclosures, confidentiality and consistency with what was actually executed.

The 507 tasks span five fictional organizations: retail and e-commerce (98), travel and hospitality (104), auto insurance (100), neobank internal IT (104) and consulting IT/HR support (101). Every model was run 20 independent times on every task.

#Once versus every time

Retries flatter almost everyone. GPT-5.4 finds a working trajectory at least once on 91.12% of tasks but repeats it on a quarter of them; Qwen3.8-27B reaches 89.35% pass@20 and 7.50% pass^20. Grok-4.3 never passed all 20 trials on a single task. The authors' point is that pass@20 measures discovery and pass^20 measures dependability, and that only the second is what a business workflow needs.

#Plausible failure is the norm

The most uncomfortable number is in the evaluator ablation. Across 79,853 failed trials, 80.88% both terminated cleanly and invoked a state-changing tool, and 67.24% also ended without an explicit error in the final tool response. A response-level or tool-call-level grader would have marked most of these runs complete.

The trace analysis says where the runs actually break. Tool-usage failures — a tool error, a failed precondition or an unsuccessful lookup the agent never recovers from — account for 79.9% of failed traces on average, and 96.4% for Claude Opus 5. Wrong state updates account for 9.4%, incomplete user resolutions 7.5%, and never attempting the required mutation only 3.2%. Agents mostly get far enough to try the work and then fail to read what the environment told them.

Figure 5: per-domain failure rate (100 − pass@1). Credit: Li et al., arXiv:2608.19741.

#Why it matters

Aggregate rank hides where an agent will break. Averaged over models, retail is the easiest domain at 55.79% pass@1 and auto insurance the hardest at 31.10%. Claude Opus 5 leads four of five domains but drops to 49.95% on booking, where GPT-5.4 leads at 68.13%; Claude Opus 4.6 scores 74.90% on retail and 14.65% on auto insurance. Parameter count explains little — Qwen3.8-27B beats DeepSeek-V4-Pro, Kimi-K2.6 and GLM-5.1 on average.

The caveats are the ones any sandbox carries: synthetic records, LLM-simulated users, fictional companies, and no evidence about live production systems. What the release does give engineers is a reusable harness — microsoft/thinkingbox for the runtime and thinkingbox-data for the tasks and MCP servers — that supplies the same verdict as an evaluation metric and as an RL reward. If you are shipping an agent that touches a real record, the number to quote internally is not pass@1.