Generalization Dynamics of LM Pre-training
Abstract
People typically assume that LMs stably mature from pattern-matching parrots to generalizable intelligence during pre-training. We build a toy eval suite and show this mental model is wrong: throughout pre-training, LMs frequently and suddenly hop between parrot-like and intelligence-like computations. We call this mode-hopping. Across our suite, LMs suddenly latch onto memorized or in-context patterns instead of in-context learning, use System 1 instead of System 2 thinking,\[p5\] pick up what sounds true instead of what is true,\[p4\] fail at multi-hop persona QA, out-of-context reasoning, and emergent misalignment -- then just as suddenly revert and generalize. Mode-hopping is not explained by standard optimization dynamics: it is locally stable and cannot be fixed by checkpoint averaging.\[p7\] We instead think of it as a capacity allocation problem: in a capacity-bounded model, generalizable circuits must compete with the shallow ones learned early in training, and the data in each pre-training window may decide which circuits win. Our suite provides a new efficient lens on generalization. We demonstrate two concrete applications: (i) select intermediate pre-training checkpoints that strongly generalize reasoning and alignment, better than the final pre- or mid-training checkpoints,\[p8\] and (ii) select pre-training data that controls and stabilizes generalization dynamics.\[p9\]
View more
View Paper128
Save
AI Overview
CopyEdit
During pre-training, language models are often judged by smoothly falling training loss and improving scores on familiar benchmarks. Those trends can conceal abrupt changes in how a model answers. A model may follow a tempting pattern in a prompt rather than infer the task, then switch back a nearby checkpoint later. On an "answer+1" arithmetic prompt, OLMo3-32B scores 81% at 2.17T pre-training tokens, 0% at 2.19T, and 81.7% at 2.21T.\[p2\] On this eval, a success means the answer consistent with the arithmetic beats the tempting successive-pattern answer. The authors call this behavior mode-hopping. The arithmetic reversal is one task-specific illustration, and it does not show that every kind of generalization reverses this way.
Schematic. Top row: pre-training loss, non-emergent downstream capability, and emergent downstream capability. Bottom row: three generalization types.
[](https://www.alphaxiv.org/abs/2609.33150#a-behavior-not-a-score "Jump to section")A behavior, not a score
Here, generalization means inferring the task or transferable structure rather than following a shallow cue. The authors built a suite of six cheap, zero- or few-shot behavioral tests, plus two fine-tuning generalization tests. The tests contrast:
- flipped labels versus familiar sentiment patterns;
- repeated or successive prompt answers versus solving the presented problem;
- truth versus truthiness;
- intuitive wrong answers versus reflective reasoning;
- disconnected facts versus a coherent persona.
- They find mode-hopping in hard accuracy and also in the soft probability margin P(correct)−P(incorrect)P(\\text{correct}) - P(\\text{incorrect})P(correct)−P(incorrect).
- They find it when plotting against pre-training FLOPs as well as against tokens.\[p6\]
- randomly sampled data (the uncontrolled run);
- subsets chosen because earlier windows favored pattern-matching on the answer+1 eval;
- subsets chosen because earlier windows favored generalization on that eval.
In the arithmetic case, the demonstrations yield 1, 2, 3. The test arithmetic answer is 8, and the tempting continuation is 4.
The prompt-based tests compare probabilities of predefined answer spans instead of relying on the model to generate extractable answers. Their results are averaged over four random seeds.\[p3\] Multi-hop persona QA works differently because it is generative. It has only five test questions per persona, which are sampled repeatedly.
The study tracks general pre-training checkpoints of OLMo3 and Apertus and excludes mid-training and long-context stages. The authors report that these models had trained for 9×–90× Chinchilla-optimal budgets.\[p2\]
[](https://www.alphaxiv.org/abs/2609.33150#why-a-reversal-is-not-just-noise "Jump to section")Why a reversal is not just noise
Schaeffer et al., in "Are Emergent Abilities of Large Language Models a Mirage?", argue that discontinuous or nonlinear evaluation metrics can make smooth output changes look like sudden emergence. Wen et al. rule out that simple reading in two ways:
Common benchmark performance, by contrast, stays stable across pre-training.\[p6\]
A second check tests whether single updates cause the hopping. The authors take one OLMo3-32B optimization step on randomly sampled pre-training data, varying batch size and learning rate. Suite probabilities change negligibly, even at learning rate 1e-2.\[p7\] Averaging five checkpoints along an oscillating curve mitigates mode-hopping but leaves it in place.\[p7\]
These tests argue against a simple explanation in terms of local optimization fluctuations, but they do not identify the cause. The results are also dataset-specific: the average correlation across dataset pairs is usually low, though larger models have higher correlations.\[p7\]
[](https://www.alphaxiv.org/abs/2609.33150#checkpoints-and-data-as-levers "Jump to section")Checkpoints and data as levers
Using their toy suite, the authors select OLMo3-32B checkpoints at 4.5T and 4.9T tokens. After math SFT, the 4.5T checkpoint scores 36.3% on GPQA versus 29.8% for the 4.9T checkpoint. After general post-training, its robustness to prefilling attacks is 53% versus 21%.\[p8\] The 4.5T checkpoint also beats the other sampled checkpoints in this comparison, even though later pre- or mid-training can improve in-distribution performance.\[p9\] Under these post-training setups, an intermediate checkpoint can be a better starting point. The results do not show that earlier checkpoints win in general.
Top: fine-tuning step against MATH-500 accuracy, GPQA-Diamond accuracy, and robustness to prefilling attack, comparing 4.5T and 4.9T checkpoints. Bottom: the corresponding measures against pre-training tokens (T), marking the selected high/low in-context-learning checkpoints.
A preliminary, small-scale experiment on one eval tested whether training data can steer this behavior. Starting from an intermediate OLMo3-32B checkpoint, the authors continued training on three kinds of data:
The controlled runs stabilize toward their intended behavior, while the uncontrolled run hops.\[p9\]
The authors interpret these findings as competition between shallow and generalizable circuits in capacity-bounded models.\[p2\] This is their hypothesis. The experiments demonstrate behavioral changes, not the circuits themselves.
[](https://www.alphaxiv.org/abs/2609.33150#check-the-next-checkpoint "Jump to section")Check the next checkpoint
Generalization is not guaranteed to mature steadily with more training.\[p1\] The suite is a cheap behavioral monitor that reveals reversals. In the reported applications, it helped choose a checkpoint and, with data selection, steer one targeted behavior. The suite is available in a GitHub repository.
Audio
0:00\--:--
1x
Transcript
Similar papers
Discussion
Comment