MazeBench: the best coding agent collects 13 gems, four frontier models get zero

A fresh run of MazeBench leaderboard results puts most of the frontier on the floor. In the code-enabled track, grok-4.6, ox-alpha, glm-5.3 and qwen3.8-max all scored 0%; gemini-3.7-flash and Kimi K3 managed 1%. The best result on the board, GPT-5.6 Sol running in Codex at max reasoning, is 13% — meaning it found 13 of the benchmark's 100 hidden gems.

The MazeBench leaderboard as posted by the benchmark's author. Credit: @patience_cave.

Gemini 3.7 Flash and Kimi K3 both round to 1%, with Gemini's bar marginally ahead — the distinction the author drew when posting the chart. Two entries are worth noting together: ox-alpha, the stealth model Z.ai confirmed as GLM-5.3-Flash, and glm-5.3 itself both sit at 0%.

#What MazeBench actually asks for

MazeBench is a 3D open world built by Jonathan and David Pappas, introduced in July and sponsored by Prime Intellect's residency program. It hides 100 gems across more than 200 rooms filled with Sokoban-style box-pushing puzzles, elevator tiles, toggleable walls and icy slopes. One gem equals one percent; a separate exploration score tracks rooms visited. Agents get eleven MCP actions — four movement, four camera, undo, level reset and teleport to any previously visited room — and the defining mechanic is camera rotation: the world can be viewed from twenty angles, and movement commands rotate with the view. Models play inside their native harnesses (Codex, Claude Code, Grok Build, Antigravity, Prime Agent) so long runs stay affordable, and there are vision, ASCII and JSON observation tracks.

The critical split is Python. With Python, coding agents can reverse-engineer the physics and write solvers, which is how the top three reach 11–13%. Without it, the author reports no model has scored above 1% — while humans can beat the environment without Python at all. The 0% and 1% rows above are on the code-enabled board, which makes them harsher than they look: those agents had solver access and still collected no gems.

#The failure modes are specific

The write-up is less a scoreboard than a bestiary of how agents break down. Older models descend into what the author calls the "Valley of Stagnation" — wandering in circles without producing new board states, tracked by a moving-average novelty chart. GPT-5.5 and Opus 4.8 both stall on the same two two-step tool puzzles needed to escape the first four rooms, each coming within one move of the solution. GPT-5.6 found a hidden room, walked away, spent hundreds of moves apparently doing nothing, then returned and executed a roughly hundred-move plan almost perfectly without Python — a puzzle the author says took him thirty minutes. Fable 5 learned to use boxes as hooks to pull others, which GPT-5.6 never managed.

Even the leaders explored only about 25% of the world, and each burned roughly a week's worth of token allowance on a max subscription. Some runs exceeded a billion cumulative tokens, which is why the leaderboard normalises on uncached input tokens. All of them handled the flat 2D section and "lost all understanding" once rooms became genuinely three-dimensional; Sol spent 5,000 moves circling one icy room before giving up.

#Astra breaks the no-Python ceiling

The board has since moved. The author reports that GPT-6 Astra scored 14% without Python after a run of more than 60 hours, against 2% for Claude Fable 5.1 and 1% each for Fable 5, Opus 5 and GPT-5.6 Sol on the same track.

The no-Python track after GPT-6 Astra's run. Credit: @patience_cave.

That is a solver-free score above what any coding agent has managed with a solver. Per Rohan Paul's summary, Astra planned 10–20 moves ahead in batched actions, cutting a projected 3-billion-token trajectory to roughly 350 million — and its remaining failures were on the genuinely 3D puzzles, suggesting the jump is in sustained planning rather than spatial understanding.

#Why it matters

MazeBench is not saturated, not close, and cheap to interpret: a gem is a gem. It separates models that can hold a hundred-move plan from models that can write a script, and it does so on an axis — embodied 3D spatial reasoning over long horizons — where the published scores are still single digits for almost everyone. A benchmark where four frontier releases score zero with code execution enabled is a useful counterweight to leaderboards that are already crowded at the top.