GPT-6 Astra takes 10 of 16 RuneBench records, at $15 a task

OpenAI's GPT-6 Astra has taken over RuneBench, the benchmark that scores AI coding agents on how fast they can train skills inside an emulated RuneScape server. It holds the record on 10 of the 16 skill tasks, and its aggregate score — the mean of ln(1 + peak XP/min) across all 16 — is 7.26, against 6.28 for the next model, xAI's Grok 4.6. Astra is a full point clear of second place; everything from Grok down through Anthropic's Fable 5.1 at extra-high effort (6.14) to Google's Gemini 3.8 Flash (5.96) is packed inside a third of that.

RuneBench peak XP/min per skill in the 30-minute trial. Credit: Max Bittker.

Max Bittker, who built the benchmark, called it "huge outlier performance." His post labels the model GPT-5.6 Astra; the published results table and OpenAI's own naming both call it GPT-6 Astra.

#How the benchmark works

Agents do not see pixels. They write and execute TypeScript against rs-sdk, a game-control library exposed over MCP, driving a headless LostCity server running at 8x speed. Each agent gets a folder of markdown files scraped from the game wiki — skill guides, item stats, NPC locations, quest walkthroughs — and 30 minutes to train one skill. The score is not total XP but the peak XP rate in any 15-second window, a design Bittker chose deliberately: scoring on totals rewarded uninterrupted grinding, while scoring on peak rate rewards agents that go exploring for a better method and then exploit it.

#Where the records are

Several of Astra's wins are not incremental. Reading from the published run data:

It also took Attack, Defence, Ranged and Cooking. In its default-effort run it reached level 99 — the game's cap — in Woodcutting, Fishing, Firemaking and Thieving inside the 30-minute window, finishing the thieving trial on just over 6 million XP.

The six it did not win are worth reading too. Fable 5.1 keeps Strength (428 vs 224), Claude Opus 5 keeps Crafting (814 vs 758), GPT-5.6 Terra keeps Woodcutting (1,250 vs 1,080) and Grok 4.6 keeps Fishing (1,300 vs 1,220); Mining and Prayer are exact ties. Astra is not uniformly ahead, it is spectacularly ahead on a subset.

#The price of the record

Astra is also the most expensive entry on the board. Its 16 tasks cost $244 in API spend, an average of $15.26 per 30-minute trial, versus $5.11 for Grok 4.6 and $2.63 for Gemini 3.8 Flash. On a leaderboard measuring how fast an agent can optimise a virtual woodcutting route, that is a curiosity; on the real long-horizon work this benchmark is a proxy for, it is the whole question.

#What not to read into it

Bittker is candid about the limits, and the run data backs him up. Each cell is a single 30-minute trial, so noise and false negatives are baked in. The clearest illustration is Astra against itself: the two reasoning-effort configurations RuneBench ran scored 3,235 and 243 XP/min on Hitpoints, and 3,065 and 1,082 on Fletching. Same model, thirteen-fold spread. The benchmark is also explicitly a test of harness and strategy as much as of the base model — RuneBench's own mirror on BenchLM keeps it display-only for that reason — and the wiki-file setup rewards a model that reads documentation well before it rewards one that plays well.

Still, the shape of the result matches what the rest of Astra's launch benchmarks showed: a model whose gains are concentrated in operating software over long stretches rather than in one-shot answers. RuneScape happens to be an unusually legible instance of it, complete with video of every run.