A team from NYU, Modal Labs, UCLA, UIUC and Columbia has put a functional form on how pretraining and reinforcement learning interact — by giving up natural language and training models on chess instead. Understanding Reasoning from Pretraining to Post-Training, by Jingyan Shen, Ang Li, Salman Rahman, Yifan Sun, Micah Goldblum, Matus Telgarsky and Pavel Izmailov, pretrains ten models from 5M to 1B parameters on human Lichess games, fine-tunes them on synthetic reasoning traces, then runs GRPO with verifiable rewards on chess puzzles. Out of 36 pretraining–RL combinations comes a joint scaling law and evidence that RL does not merely sharpen what supervised fine-tuning already believed.
The pipeline mirrors standard LLM training — pretraining on tokenized human games, SFT on synthetic search traces, RL on a verifiable puzzle environment. Credit: Shen et al., arXiv:2607.16097.
Frontier-scale sweeps across both stages are unaffordable, and web-scale corpora make it impossible to attribute a behavior to pretraining rather than RL. Chess fixes both: a compact action space and exact ground truth at every board state. The corpus is 54B tokens of 2022 Blitz and Rapid games; post-training uses 156K quality-filtered puzzles in five difficulty bins, evaluated on 1,480 held-out tactical puzzles. Reasoning happens entirely in move tokens — traces are sampled continuations merged into a search tree and serialized depth-first, so the model searches in context rather than calling MCTS.
The paper's headline result is that a pretrained checkpoint's two most useful properties predict almost everything about the RL run that follows. Post-RL pass@1 at a fixed RL compute level is predicted by pretraining validation loss (Spearman |ρ| tightening from 0.93 to 0.99 as the reference RL compute rises from 10^16 to 10^20 FLOPs), while the slope of the RL reward curve — reward gained per decade of RL compute — grows roughly linearly with log pretraining tokens (Pearson r = +0.84), with model size a weaker positive correction.
The compute-optimal RL share climbs from about 19-20% at 50-80M parameters to 28% at 680M, while the optimal pretraining token count stays close to Chinchilla. Credit: Shen et al., arXiv:2607.16097.
Combined with a Chinchilla loss fit, that law lets the authors score recipes they never ran. The extrapolated frontier says the compute-optimal RL share rises with total budget — roughly 19-20% at 50-80M parameters up to 28% at 680M — while pretraining token counts stay near Chinchilla-optimal. Empirically, the 20M model's RL share along the frontier climbs from 5% to 32%. The flip side: RL is initialization-limited, and the extra compute bought by cutting pretraining short does not compensate for the weaker prior.
This is the passage Gavin Leech surfaced on X. Testing whether the RL policy is a temperature-scaled SFT policy, the authors find the global sharpening exponent does rise — but the fit is only moderate and per-state slopes vary widely. Categorizing every board state instead, they see three distinct behaviors: ground-truth amplification (the correct move was already in the top-3 and gets reinforced), tail discovery (a correct move sitting below 0.05 probability is promoted into the top-3), and wrong-mode amplification (the correct move stays outside the top-3 while a preferred wrong move is reinforced further).
Amplification dominates on easy puzzles. On hard ones, tail discovery and wrong-mode amplification both increase. That mixed redistribution is the paper's explanation for a familiar pattern: RL reliably improves pass@1 while pass@16 stays flat or degrades slightly for larger models. It also cuts through a live disagreement — Yue et al. argued RL only sharpens, Yuan et al. that it composes new skills — by showing both happen, sorted by difficulty.
A parallel look at the reasoning traces finds RL widens search rather than deepening it: branching factor rises while maximum depth stays flat, and models still fail on lines longer than five moves.
One language check: a 1B OLMo-2 model pretrained on 200B tokens of mostly math text, checkpointed 14 times between 10B and 200B tokens, SFT'd on NuminaMath-CoT and RL'd on 24.9K GSM8K, MATH and DeepScaler problems. The same shape appears — lower pretraining loss predicts higher post-RL pass@1, and the RL slope grows with log tokens.
Almost every public argument about where the next training dollar goes — better base model or more RL — has been made without a functional form to argue over. This paper supplies one with fitted coefficients, and its answer is neither camp's: pretraining sets both the level and the improvement rate of everything RL does afterward, so RL's share should grow with total compute without crowding out the prior. The caveats are the authors' own — a 1B ceiling, chess as a proxy, and a slope relationship they call a local trend over the studied compute range rather than a global law. Models, datasets and code are at pavelslab-nyu/pre2post-chess.
