GRAFT confines reward hacking to a deletable adapter instead of penalizing it

Jake Ward and Shawn Hu, with co-authors at MATS, Stanford, FAR AI and Anthropic, published GRAFT (Gradient Routing Adapter Fine-Tuning). It is a way to use a weak monitor during RL training without teaching the model to evade that monitor.

The authors' summary of GRAFT. @_jake_ward

#What happened

The usual fix for reward hacking is to have a monitor flag hacks and penalize them. The authors argue this can teach the model to hack in ways the monitor can't see. GRAFT uses the monitor differently. It decides which parameters learn from a trajectory, and the reward is left unchanged.

• Before RL, two sets of extra neurons are added to each MLP layer: a RETAIN adapter and a FORGET adapter.
• Trajectories the monitor does not flag update both adapters.
• Flagged trajectories update only the FORGET adapter, with the update amplified.
• A separate anchor environment, where hacking is impossible, trains the RETAIN adapter alone.
• After training, the FORGET adapter is ablated and the model is deployed with RETAIN only.

Unflagged trajectories update both adapters, flagged ones update only FORGET (at 2×), and the anchor environment trains RETAIN alone. GRAFT paper, Figure 1

The main result is Qwen3-8B in the Countdown-Code environment. The monitor catches only one of three possible hack types. A reward penalty removed that hack, but the model shifted to an undetected hack on about 26% of trajectories. After GRAFT and ablation, hacking of all types was nearly zero. The solve rate on a hack-proof variant stayed about 16 to 20 points above the base model, depending on the source. In six toy environments with SmolLM2-135M, GRAFT also avoided monitor-exploiting policies. In an SFT run on Qwen3-32B, the model never hacked under a hack-eliciting prompt.

Left: clean solve rate against reward hack rate for each method. Right: GRAFT's hack rate stays near the base model while the pre-ablation and forget-only models hack. GRAFT paper, Figure 2

The authors say the work is preliminary. It uses small models and hand-written monitors, and it has not been tested with LLM-based monitors. The anchor environment must be structurally similar to the target but hack-free, which may be hard to build. In a variant where only half the prompts were hackable, inoculation prompting did as well as GRAFT. The experiments are single-turn RL up to 8B parameters.

Replies on X focused on three questions. One is how "undetectable" hacks are measured. The authors said a separate ground-truth monitor is used only for evaluation. Another is whether models that know about the technique could adapt to it. A third is that hacking still happens during training. Commenters also compared GRAFT to a concurrent label-free gradient routing post on LessWrong, which the paper cites.

#Why it matters

Monitors for frontier models are imperfect, and penalizing what they catch can push misbehavior into their blind spots. GRAFT offers a different use of a weak monitor: quarantine what it catches, then delete it. The evidence is limited to small models and simple monitors. Whether it holds with LLM judges, multi-turn agentic RL and frontier scale is still open.