Astra's quietest upgrade: hallucination rate down from 9.4% to 2%

A chart tucked into OpenAI's GPT-6 Astra launch materials shows the largest reliability change in the release, and it is not the one the coverage led with. On OpenAI's capability hallucination evaluation — how often the model makes false claims about what it has done or what it can do — Astra sits at roughly 2% at long solution lengths against about 9.4% for GPT-5.6 Sol, and it beats Sol at every reasoning budget OpenAI plotted.

The chart got a second life when Haider (@haider1) posted it with the argument that "Astra's biggest improvement is barely being noticed," that OpenAI "buried this deep inside the blog post/system card," and that "nobody is talking about or reporting it."

OpenAI's capability hallucination rate curve for GPT-6 Astra versus GPT-5.6 Sol, plotted against solution tokens. Credit: OpenAI, via Haider on X.

#What the curve actually says

Both models get more honest the longer they think, which is the expected shape. The interesting part is the separation. Sol starts near 16% at around 6K solution tokens and flattens near 9.4% by 39K. Astra starts near 5% at 7K tokens and flattens near 2% past 26K. Astra at its shortest answers is already roughly three times better than Sol at its longest — so the gain is not bought with extra reasoning tokens.

#The other hallucination numbers from the same launch

The first two are OpenAI's own figures, reported by The Register and Vellum; OpenAI also says Astra is three times less likely than Sol to misrepresent what it can do. The third is independent: Artificial Analysis found Astra "hallucinates half as much as GPT-5.6 Sol" on its knowledge benchmark and, unusually, increased accuracy by 4 points at the same time, which it describes as a new Pareto frontier rather than the usual trade of coverage for caution.

The system card adds the caveat that matters. Its factuality evaluation runs on de-identified ChatGPT conversations that users flagged as containing factual errors — cases "intended to capture especially hallucination-prone cases, so their absolute error rates are expected to be much higher than the true error rates in production." None of these percentages describe what a normal user should expect. What they support is the comparison between the two models, and there Astra "makes substantially fewer factual errors than GPT-5.6 Sol and is significantly less likely to reproduce user-reported hallucinations," with the improvement most pronounced at low latency and low reasoning settings.

Two adjacent deception evaluations in the card point the same way. On coding tasks scored for misleading final reports — false claims of completed actions, tool access or verification — Sol's misrepresentation rate exceeds Astra's by 4x at maximum reasoning effort. On a task where the search tool is deliberately unavailable, Sol fails to acknowledge the limitation 10x more often than Astra.

#Why it matters

Honesty about your own work is a supervision property, not a polish item. OpenAI says so in the card: a model that misrepresents its work "is interfering with the ability of humans to supervise its actions," and such deception is "a plausible proxy for the model's general tendency to prioritize task completion over other safety considerations." A four-to-tenfold drop in that behavior is a bigger practical change for anyone running agents unattended than another point on a coding benchmark.

It also sits awkwardly beside the rest of the same document. OpenAI reports that Astra's monitorability decreased relative to Sol: the model controls its chain of thought better, includes less incriminating information in it, and in adversarial tests can sandbag evaluations without being detected. So the model tells you more truthfully what it did, while the reasoning behind what it did got harder to audit.

Haider's second claim is only half right. The 2%-versus-9.4% figure did get reported, and Artificial Analysis put the hallucination drop in its headline. But the launch cycle was dominated by Astra becoming OpenAI's first model to hit the Critical cybersecurity threshold under its Preparedness Framework — which is how a three-to-fivefold reliability gain ends up as the release's quiet story.