A read-only activation probe can leave a trace a language model learns to recognize. In an October 18 research thread, Sauers demonstrates this with Qwen3-8B: identical prompt tokens and model weights produce different answers depending on whether an internal activation is being saved. The signal comes from how the software executes the model.
Identical inputs, different execution conditions. These responses were deliberately trained. Figure: Sauers / Watching.
The training setup keeps the first 34 blocks frozen and fine-tunes the final two blocks; the technical report also specifies training the final normalization. A probe reads layer 14. Supervised examples pair the same question with probe-on and probe-off runs.
The detector is learned in the model’s final layers. Figure: Sauers / Watching.
The implementation comparison explains the apparent paradox. Saving an activation can force the compiler to write an intermediate result to GPU memory and split operations into separate kernels. That changes the order of arithmetic and rounding, leaving small numerical differences downstream. “Read-only” at the Python level therefore does not guarantee identical compiled computation.
Three variants help separate the effects. The simple probe is readily learned. Splitting the computation into two kernels without a probe is also learnable, but takes longer. A probe that copies into a pre-allocated buffer keeps the step in one kernel; that variant changes some activations, but detection was not learned with this recipe.
Probe implementation changes both the numerical signal and how readily detection can be learned. Figure: Sauers / Watching.
The thread’s circuit analysis traces the effect through the early user token. Its hidden state contains a consistent numerical difference between conditions. Copying that state from a probed run into an unprobed one reproduces almost all the answer change; interventions identify head 6 in the final layer as the main route to the output.
Interventions connect the numerical difference to the answer switch. Figure: Sauers / Watching.
Training amplifies the signal by shrinking the background component of an attention key. A small perturbation then produces a large change in its normalized direction, redirecting attention toward the token that pushes the answer to Yes.
The proposed mechanism links compiler behavior to a learned attention circuit. Figure: Sauers / Watching.
Separate demonstrations distinguish execution environments and write a message into hidden states. In the latter, a logit lens decodes “I can see you watching me” from the compiled model. The message was a training target; it is not a spontaneous confession.
Two separately trained behaviors extend the idea beyond the original probe. Figure: Sauers / Watching.
The report records 128/128 correct probe-detection answers on 64 new prompts on the training GPU, an L40S. Transfer weakens on other GPUs and reverses on H100 and A100. System-first chats also fail in the tested setup. The training-versus-deployment experiment detects differences between runtimes: disabling gradient recording in the same Hugging Face execution does not itself supply the claimed distinction.
The lower panel concerns probe detection; the upper panels concern a separate gradient-recording detector. They are different experiments. Figure: Sauers / Watching.
This is a released experimental report with code, data and model checkpoints. It establishes a learnable implementation signal under particular conditions. Whether ordinary deployed models acquire such behavior without targeted training remains an open question.
Watching’s own related-work section cites FloatDoor: Platform-Triggered Backdoors in LLMs, which studies trained behavior triggered by numerical differences across deployment platforms. Both studies exploit execution-dependent arithmetic; Watching focuses on activation capture and runtime distinctions.
Primary sources: Original thread · Complete thread and figures · Watching report, code and results.
Sauers: activation-probe detection threadWatching: complete thread and figuresWatching: technical report, code and resultsFloatDoor: Platform-Triggered Backdoors in LLMs
