Table of contents

1. Introduction

Assume labs have already trained their largest frontier language model, but the model is too large for efficient inference. Our goal is to compress it into a smaller model to reduce inference cost while preserving as much capability as possible. One obvious way of doing this is knowledge distillation, and every labs are already doing so to train their small language models. However, the details are rarely disclosed. Does distillation here mean training on synthetic data generated by the teacher? Or does it mean logit distillation, which uses teacher logits to set the target probabilities instead of one-hot labels?

Importantly, in practice labs have access not only to the final post-trained model, but also to all of its pre-training checkpoints, as well as intermediate post-training checkpoints such as the SFT and RL models. These artifacts from the frontier model’s training trajectory carry rich information and should be leveraged when training the compressed model.

Here we specifically study whether logit distillation during pre-training improves student model performance, rather than synthetic data, whose search space is too vast. We share two observations:

(§3.1) Distillation pre-training does not improve data efficiency, but may improve compute efficiency: under data-constrained setups, distillation pre-training has a higher loss asymptote than vanilla pre-training. However, distillation pre-training is more compute-efficient than vanilla pre-training in certain setups (iso-data student training or when the teacher-student gap is large enough).

(§3.2) Selection of teacher model matters for downstream performance: distilling from a teacher checkpoint trained with SFT improves the student’s SFT val loss more than distilling from a plain pre-trained teacher, and distilling from a teacher checkpoint trained with RL improves the student’s final RL performance more than distilling from an SFT teacher and pre-trained teacher.

Based on these observations, we suggest the following:

  1. Use distillation pre-training for a compute-efficiency win (but account for the compute cost of teacher inference and training); if you specifically want a data-efficiency win, try distillation annealing instead of a fixed distillation weight throughout training.
  2. Choose the teacher checkpoint based on your goal: if you have a specific downstream task in mind, use the teacher that was trained for it.

2. Experimental Details

We adopted modern LM training techniques so that our findings are relevant to frontier LM training recipes.

  • Data: DCLM for pre-training and the Tulu3 mixture for SFT. #tokens for pre-training follows the Chinchilla-optimal ratio when pre-training the teacher model.
  • Optimizer: Muon with weight decay (0.1 for pre-training, 0.0 for fine-tuning). We follow Kimi’s implementation details for adjusting the RMS of gradient updates.
  • Hyperparameters: we use muP for LR scaling and weight initialization. We use a seqlen of 4096 and a batch size of 64 for both pre-training and SFT.
    • For pre-training, we search LR on small-sized LMs and transfer it to larger model training.
    • For SFT, we use 1e-4 for the teacher and sweep {1e-4, 3e-4, 1e-3, 3e-3, 1e-2} for student models, selecting the best LR for each student.
    • For RL, we use an LR of 3e-7 for both the teacher and the student. We use GRPO with 8 rollouts per prompt, a KL coefficient of 0.001, and no entropy bonus. We apply importance sampling to correct for off-policyness.
  • Model: decoder-only transformer, using QK norm, pre-LN, and tied embeddings. We use a vocab size of 1024 to reduce the size of the embeddings unless stated otherwise.
  • LR scheduler: for pre-training and SFT, we use WSD without warmup, linear decay for the final 10% of steps. For RL, we use constant LR without warmups.
  • Data replay for SFT: we use a 1:9 mixing ratio of DCLM and SFT data during the SFT stage to mitigate forgetting.

For all distillation experiments, we fix \(\alpha = 1\) in \(\mathcal{L} = (1 - \alpha) \cdot \mathcal{L}_{\text{CE}} + \alpha \cdot \mathcal{L}_{\text{KD}}\), i.e., we train purely on the distillation loss and drop the CE term entirely. We have not tuned \(\alpha\), so there is likely further improvement available from mixing in some CE loss rather than using distillation loss alone.

3. Main Results

3.1. Distillation pre-training does not improve data efficiency, but may improve compute efficiency

Viewing logit distillation as a form of implicit regularization from the teacher, we were curious whether it improves data efficiency during pre-training. For this experiment, we adopted a data-constrained setup: we fix the amount of pre-training data (200M, 400M, 800M, or 6.4B tokens) and train the teacher at the Chinchilla-optimal ratio. We then distill into a smaller student model (1/4 or 1/16 the teacher’s size), repeating the data for up to 8 epochs in each configuration. Following Kim et al., 2025, we run coordinate descent to find the optimal LR and weight decay for each configuration, then fit a power law \(y = A x^B + C\) to find the asymptote for each configuration.

The figure below shows the result. The x-axis represents the number of training epochs for the student; N epochs means the student is trained for N epochs over the amount of data used for teacher training.

Multi-epoch results with a compute-optimal ("no epoching") teacher across six data/size configurations. Solid curves are fitted power laws; dotted lines are the fitted asymptotes for vanilla pre-training (sup) and distillation pre-training (sup KD), and the dashed line is the teacher.

The results show that in every case, the sup asymptote (=vanilla pre-training) has a lower loss than the sup KD asymptote (=distillation pre-training), meaning that distillation pre-training does not give a clear data-efficiency win. Furthermore, distillation pre-training does not give a compute-efficiency win in every setup, especially in over-training scenarios with a small teacher-student gap (upper side of the figure: for epochs 2, 4, and 8, vanilla pre-training has a lower loss than distillation pre-training). This finding resonates with the observation in Distillation Scaling Laws (Busbridge et al., 2025) that, given sufficient compute, the vanilla training objective can be more compute-efficient. The difference from their setup is that ours is data-constrained, whereas theirs is not.

In the previous plot, the teacher is always trained with a compute-optimal setup. To probe whether a stronger teacher helps, we instead trained the teacher for 16 epochs over the same data (which improves its validation loss) and then distilled from this teacher. Results are shown below: although using a stronger teacher improves the asymptote of the distilled student’s loss, distillation pre-training is still not clearly more data-efficient than vanilla pre-training. On 4 out of 5 configurations, training with distillation loss matches with training with vanilla cross-entropy loss.

The same configurations, now distilling from a teacher trained for 16 epochs over the same data. The KD asymptotes improve relative to the compute-optimal teacher above, but still do not fall below the vanilla asymptotes.

Explanation: Distillation loss reduces the gradient variance but introduces teacher bias

A simple bias-variance argument predicts exactly this shape. Write \(p_\theta(\cdot \mid x)\) for the student’s next-token distribution at context \(x\), \(q_T(\cdot \mid x)\) for the teacher’s, and \(e_y\) for the one-hot vector of the observed token \(y\). Vanilla and distillation training minimize the same cross-entropy loss, just against different targets:

\[\mathcal{L}_{\text{CE}}(\theta) = \mathbb{E}_{X,Y}\big[-e_y^\top \log p_\theta(\cdot \mid x)\big], \qquad \mathcal{L}_{\text{KD}}(\theta) = \mathbb{E}_X\big[-q_T(\cdot \mid x)^\top \log p_\theta(\cdot \mid x)\big].\]

Let \(J_\theta(x)\) be the student’s logit Jacobian at \(x\); the per-token gradients likewise differ only in the target:

\[g_{\text{CE}}(x, y) = J_\theta(x)^\top\big(p_\theta(\cdot \mid x) - e_y\big), \qquad g_{\text{KD}}(x) = J_\theta(x)^\top\big(p_\theta(\cdot \mid x) - q_T(\cdot \mid x)\big).\]

Conditioned on \(x\), the one-hot target \(e_y\) is a random draw from the true conditional \(\pi(\cdot \mid x)\) while the teacher target is deterministic, so distillation removes the next-token sampling noise: \(\mathrm{Cov}[g_{\text{CE}} \mid x] = J_\theta^\top(\mathrm{Diag}(\pi_x) - \pi_x\pi_x^\top)J_\theta \succeq 0\) but \(\mathrm{Cov}[g_{\text{KD}} \mid x] = 0\). That is the variance win, and it is why one soft target can be worth many sampled tokens.

An example for explaining variance

Take the prefix “She poured herself a cup of”, where the next word is spread over several candidates: tea 0.45, coffee 0.30, water 0.15, juice 0.05. The corpus shows this context once with whichever token the author wrote; if it is coffee, CE calls the other three wrong, and a different draw would have pointed the gradient elsewhere. On the other hand, a teacher returns the whole ranking in one pass, e.g. tea 0.40, coffee 0.33, water 0.17, juice 0.06, and returns it unchanged every time.

But the two objectives have different population minimizers: \(\theta^\star = \arg\min_\theta \mathcal{L}_{\text{CE}}(\theta)\), the best student approximation of the true distribution, and \(\theta^\dagger = \arg\min_\theta \mathcal{L}_{\text{KD}}(\theta)\), the best student approximation of the teacher. At \(\theta^\star\) the distillation gradient \(\nabla \mathcal{L}_{\text{KD}}(\theta^\star) = \mathbb{E}_X[J_{\theta^\star}(x)^\top(\pi(\cdot \mid x) - q_T(\cdot \mid x))]\) is nonzero — the teacher’s error projected onto directions the student can express — so distillation pulls the student off the true optimum by a fixed amount \(\delta_{T,S} = \mathcal{L}_{\text{CE}}(\theta^\dagger) - \mathcal{L}_{\text{CE}}(\theta^\star) \ge 0\) that does not shrink with more training.

Our explanation rests on the same statistical mechanism Menon et al., 2020 identify in classification: an optimal (Bayes) teacher lowers the variance of the student’s objective at no cost in bias, and an imperfect teacher trades that variance reduction off against how poorly it approximates the true class-probabilities. Our version differs in what it compares — the population minimizers that gradient descent converges to under each loss, rather than the deviation between the distilled and the population risk for a fixed predictor over \(N\) i.i.d. samples.

Achieving best of both worlds: distillation annealing

Based on this explanation, we suggest using distillation annealing: keeping the distillation loss on early captures the variance reduction, and decaying it to zero removes the bias term in the limit. We ran a preliminary annealing experiment on top of the 16-epoch teacher, which weights distillation heavily at the start of training and linearly anneals it down to zero by the end. This does improve the asymptote: specifically, in the 800M-token, 1/4-params setting the KD asymptote with annealing reaches 1.942, below the vanilla asymptote of 1.970.

Adding annealing to the 16-epoch teacher, for the three 1/4-params configurations.

We later found that Rawat et al., 2024 had a similar idea but in the reverse direction: their SALT distills from a smaller 1.5B teacher into a larger 2.8B student, running KD only for the first 36K of 208K pre-training steps and then switching to plain next-token prediction.

Thoughts: why this contrasts with the self-distillation result in Kim et al., 2025

Our conclusion sounds like it contradicts Kim et al., 2025, whose data-constrained setup and asymptote-fitting methodology we borrow. They distill a 300M teacher into a freshly initialized student of the same architecture and get a student that beats the best regularized 300M model trained directly on the same tokens — a data-efficiency win from distillation, which is exactly what we fail to find.

Figure 9 of Kim et al., 2025; self-distilling a 300M teacher into a same-architecture 300M student (green star) reaches 3.43, beating the best single regularized 300M model at 3.586 and matching the asymptote of the regularized parameter-scaling recipe, while distilling an 8-ensemble reaches 3.36.

Our result differs from Kim et al. primarily because the two studies operate in different capacity regimes. Our students are 4–16× smaller than a compute-optimal teacher and see many unique real tokens per parameter (N/D=80-320), which makes the optimization problem hard. Kim et al. study the opposite regime: a 300M model trained from only 200M real tokens, which they explicitly identify as overparameterized and prone to overfitting. Moreover, their self-distillation is not purely learning the real data, and instead, it mixes real data with 10B generated teacher tokens. Their ablation shows that synthetic-only self-distillation is worse than the teacher, while mixing synthetic and real data allows the student to improve. Their gain is therefore best understood as reducing estimation error through teacher-generated data augmentation and regularization.

In short, Kim et al. show that sequence-level self-distillation can improve an overparameterized, data-limited student by augmenting the original real data. Our experiments show that pure logit distillation on fixed contexts does not improve the asymptotic loss of a smaller, capacity-limited student, although it can improve early optimization by reducing target noise.

3.2. Selection of teacher model matters for downstream performance

In short, we should use the teacher model after SFT (or RL) if we care about the eventual SFT (or RL) performance of the student model.

People care about the final LM’s performance after post-training, and existing literature already mentions the issues of selecting pre-trained checkpoints based solely on pre-training loss [Wen et al., 2026, Chen et al., 2026, Watts et al., 2026]. In other words, there might be cases where the pre-training val loss is not the best, but the final performance after SFT is better.

Hypothesis: Using an SFT-trained teacher for logit distillation during pre-training aligns the student toward the SFT distribution, improving SFT validation loss and reducing forgetting during the later SFT stage.

To check this hypothesis, we pre-trained a 350M teacher model on 6.4B DCLM tokens and then did SFT on it using the Tulu3 SFT mixture (260M tokens, one epoch of training). We used this “SFT teacher (teacher after SFT)” for logit distillation during pre-training. For ablations, we also tested PT teacher distillation (using the teacher right after pre-training, before SFT) and a no-distillation baseline. For the student, we used a 22.6M-parameter model (1/16 the teacher’s size).

SFT-teacher distillation does not give the lowest pre-training val loss

First, the plot below shows the 22.6M student pre-training results. Distilling from the SFT teacher gives a worse PT loss (2.082) than distilling from the PT teacher (2.056), though it is still slightly better than normal PT (2.085). This is because the teacher’s PT loss degrades after SFT: the teacher forgets some of the pre-training distribution during SFT, despite the pre-training data replay we use to mitigate this.

Pre-training a 22.6M student under three regimes: SFT teacher distillation, PT teacher distillation, and normal PT. Left: gradient norm. Middle: training loss (distillation or CE, so the curves are not directly comparable across regimes). Right: held-out CE loss.

SFT-teacher distillation improves compute efficiency for SFT loss

Next, we ran SFT on the student models using the Tulu3 SFT mixture, repeating the same teacher SFT data for up to 16 epochs (we stopped there because further repetition gave saturating performance). We report both Tulu3 val loss (target performance) and DCLM val loss (forgetting).

SFT-teacher distillation reaches a lower target val loss for a given number of epochs, which tells us it gives a compute-efficiency win. However, the gap between the different pre-training teachers shrinks as we train for more SFT epochs.

SFT after pre-training distillation, with a 22.6M student and a 350M teacher. Left: Tulu3 val loss (target performance), where SFT teacher distillation reaches 1.0256 against 1.0260 for the PT teacher and 1.0390 for vanilla PT. Right: DCLM val loss (forgetting).

We further confirmed these findings at two larger scales, with consistent results:

  1. A larger-scale teacher (1.1B params) and student (69M params): the teacher has 2x the layers and 2x the MLP hidden dim, pre-trained on 25.6B DCLM tokens.
  2. The same larger-scale teacher/student pair, but with a larger vocab (the Llama 3 tokenizer, >100K tokens) instead of our default 1024-token vocab.
Scale (1): 69M student, 1.1B teacher. SFT teacher distillation reaches 0.9162 Tulu3 val loss against 0.9178 for the PT teacher and 0.9310 for vanilla PT.
Scale (2): the same 69M student and 1.1B teacher with the Llama 3 tokenizer (>100K vocab). SFT teacher distillation reaches 1.8354 against 1.8594 for the PT teacher; here vanilla PT is close behind at 1.8372.

RL-teacher distillation improves final RL performance

It has now become common practice to improve LM capabilities through RL, e.g., for reasoning. However, it remains unclear how applying RL to the teacher affects logit distillation in pre-training. The common intuition is that RL sharpens the model distribution, which may reduce output diversity. This, in turn, could degrade the teacher’s predictions on the pre-training distribution and hurt logit distillation.

In this experiment, we use the synthetic reasoning dataset from Zhang et al., 2025 for SFT and RL. Specifically, we use #op (=difficulty; higher means more difficult) of 7 to 11 for SFT, and 12 to 16 for RL. We used 25.6B DCLM tokens for pre-training, a 1.1B-parameter teacher, and a 69M-parameter student. We trained for 2K steps in the SFT stage for both the teacher and student, as a larger number of steps led to overfitting.

A sample from the synthetic reasoning dataset: a problem statement, the question, a step-by-step solution, and #op as the difficulty label.

First, SFT results show that using the RL teacher for pre-training gives the best SFT validation loss (0.1189), and using the SFT teacher is next (0.1192), although they are not significantly different. Next, we performed RL with different SFT models, and it turns out that using the RL-teacher-distilled model gives significantly better training dynamics and final RL performance.

SFT stage on the synthetic reasoning data, comparing students pre-trained with vanilla PT, PT teacher distillation, SFT teacher distillation, and RL teacher distillation. Left: reasoning training loss. Middle: held-out DCLM loss (forgetting). Right: held-out reasoning loss (target performance).
RL stage starting from each of the SFT'd students, with the teacher's own RL run shown for reference. Left: pass@1. Middle: pass@8. Right: rollout entropy. The RL-teacher-distilled student tracks the teacher most closely and ends highest on pass@1.

Explanation: Choosing the teacher is choosing the direction of the bias

The basic picture is that the post-trained teacher’s logits encode which continuations are closer to the final desired policy. Even when the pre-training data contains nothing resembling the post-training distribution, the teacher logits let the student spend pre-training learning from a distribution already tilted toward the eventual target behavior, instead of first fitting the raw pre-training distribution and only later being corrected by SFT/RL.

This is the same teacher-bias term that we pay for in Section 3.1. Distillation converges to \(\theta^\dagger\), the best student approximation of the teacher, displaced from \(\theta^\star\) by the projected teacher residual \(\mathbb{E}[J^\top(\pi - q_T)]\). Against the pre-training distribution \(\pi\) that displacement is pure loss, and we measured it: the SFT teacher gives 2.082 pre-training val loss against 2.056 for the PT teacher. However, when the eventual evaluation is the post-training objective, the same displacement points at the target instead of away from it.

Additional result: using intermediate pre-training checkpoints of the teacher

In short, we tried using intermediate checkpoints from the teacher’s pre-training run as distillation targets, but the results do not support it as a reliable strategy.

The motivation is simple. When the student is much smaller than the teacher, the final teacher model may be too strong or too sharp for the student to imitate effectively. In that case, using intermediate teacher checkpoints could provide a smoother curriculum: early in training, the student distills from an earlier, weaker teacher, and later it distills from a stronger teacher. This is similar in spirit to progressive distillation [Panigrahi et al., 2024].

To test this, we used a 350M teacher model trained for 24k steps in total. We saved intermediate checkpoints from this run and used them as teachers for distillation, sampling checkpoints every 1/20 of the total training steps: 1248, 2496, and so on. Below is the training log of the teacher model.

Training log of the 350M teacher whose intermediate checkpoints we harvest. Left: gradient norm. Middle: training loss. Right: held-out DCLM loss.

The results are mixed.

Students distilled from all 19 intermediate teacher checkpoints, from step 1248 (yellow) to step 23712 (purple). Left: final held-out DCLM loss (at step 24960) as a function of the teacher checkpoint step used for distillation. Middle: KD training loss, which is lower for earlier (weaker) teachers. Right: held-out DCLM loss over training, where the training-loss ordering does not carry over.

First, the KD training loss is lower when using earlier checkpoints. This is intuitive: earlier teachers are weaker, so their distributions are easier for the student to match.

The final held-out loss tells a more nuanced story. It drops sharply as the teacher checkpoint moves from step ~1200 to ~9000, then plateaus around 2.62 through the middle third of training. In the final third, the picture gets noisy: the single best final loss (2.586) comes from an intermediate checkpoint at step ~17400, beating the final checkpoint at step ~23700 (2.598), but the checkpoints immediately on either side of that dip are worse than the final checkpoint. So while later, stronger teachers are better on average, the ordering is not strictly monotonic, and we have no reliable way to identify in advance which intermediate checkpoint, if any, will outperform the final one.

Overall, this suggests that making the teacher easier to imitate is not enough: even though early teacher checkpoints reduce the KD training loss, they do not reliably translate into a better final val loss. The final teacher checkpoint remains the safe default, and although an intermediate checkpoint occasionally does slightly better, the gain sits within the checkpoint-to-checkpoint noise, so intermediate-checkpoint distillation does not look like a dependable lever in this setup.

4. Next Steps

Our results suggest that logit distillation in pre-training is beneficial, but only when done carefully: teacher choice, data regime, and training stage all matter. Applying these findings to actually train strong, small language models — e.g., using the Marin and OLMo open-weight training pipelines — would be a valuable next step. We also plan to develop better algorithms for distilling frontier language models, particularly in settings where we have open access to the full training history of the frontier model. For example:

Leveraging richer information about the frontier model’s training: so far, we only use logits to transfer information from the frontier model. What about its weights? Or its training dynamics?

Training the teacher to be more effective for logit distillation. Can we further train the frontier teacher model (using only a small amount of compute) in ways that make it more effective for logit distillation? Our results show that SFT-ing the teacher on the target domain improves logit distillation in terms of the final student’s SFT performance. What other levers can we pull?

Doing self-distillation or weak-to-strong distillation in pre-training. So far, we have only talked about training a smaller student model. What about using a student model that has a similar or even larger parameter size? What strategic choices can we make?

5. Acknowledgements

SJ thanks Konwoo Kim, Suhas Kotha, Suhong Moon, Sehoon Kim, Minwoo Kang, and Morris Yau for their valuable feedback and support throughout this work, as well as the other members of Tatsu and Yejin’s labs. We also acknowledge Upstage for their compute support for this work.