Stealing Hidden Reasoning Traces From Proprietary LLM APIs

Ilia Shumailov and Alexander Panfilov argue on Machine Learning Street Talk that the mechanism labs use to keep chain of thought hidden is also the way to get it out. Their paper, Stealing Reasoning Traces from Proprietary LLM APIs, targets the encrypted reasoning state providers hand back so a conversation can be resumed or forked later.

The bug, as they describe it, is deceptively simple: those blobs can be replayed across users and across sibling models. A smaller model can be asked to decrypt the trace on the provider's own infrastructure and then repeat the hidden reasoning back in plain text.

The consequences they walk through:

• Leaked private data carried inside recovered traces
• A broadly reusable jailbreak derived from the recovered reasoning
• Poisoned agent traces — forked state that a later turn inherits
• Implications for chain-of-thought monitoring, plus the responsible-disclosure process and candidate defenses

#The fine print

Both guests close by drawing a line between the jailbreaking threat they actually demonstrate and ordinary benign distillation, and by arguing for controlled experiments over sweeping claims — a notably restrained framing for a result that reads like a headline. Shumailov is a security researcher formerly at Google DeepMind, with a Cambridge PhD under Ross Anderson; Panfilov is a PhD researcher at the ELLIS Institute Tübingen and the Max Planck Institute for Intelligent Systems.

Sources: YouTube, Paper