A paper from the University of Maryland shows that on-policy distillation (OPSD), a technique for improving reasoning in language models, hides a structural problem: in its standard form it only works with custom engineering. Researchers identify the cause in an implicit parameter, β, set to 1 in the vanilla version, and make it controllable. Hence the name, β-OPSD.

Why this matters. If you train or fine-tune models with reasoning capabilities, training stability is the practical bottleneck. Vanilla OPSD promises to improve reasoning without the cost of reinforcement learning, but in practice it requires trial and error to find the right settings, and often training diverges. β-OPSD turns that hidden parameter into a dial: lower β and the model explores more, raise it and it stays anchored to the starting model. The paper shows an average gain of +5.74 points on math benchmarks with Qwen3 1.7 billion parameters, and stability improvements across the range up to 8B.

The main strength is that it avoids expensive, high-variance reinforcement learning. Instead of running thousands of RL episodes, β-OPSD converts a closed-form solution into a token-level distillation target, using a return-to-go credit to align the update of the single token with the outcome of the entire sequence.

Results are measured on math benchmarks with Qwen3 models (1.7B, 4B, 8B), not on general agentic tasks. Code and details are on the University of Maryland project page.

In detail

On-policy distillation (OPSD) is a technique where a model learns from its own outputs: it generates responses, evaluates which are better, and updates to produce the good ones more often. The idea is appealing because it doesn’t require a larger teacher model to copy from, and doesn’t require classical reinforcement learning with its sparse rewards and high variance.

The problem, until now, is that OPSD in its standard form (called “vanilla” in the paper) is fragile. To make it work requires heavy engineering: manual hyperparameter tuning, stabilization tricks, and often training diverges anyway. The paper identifies a structural reason for this fragility.

The authors show that vanilla OPSD is actually a special case of a broader family of policy optimization objectives, regularized with a KL penalty (Kullback-Leibler, a measure of how much two probability distributions diverge). In this family, the parameter β weights the KL penalty that keeps the student close to a reference model. In the vanilla version, β is implicitly set to 1, with no way to change it.

The move of β-OPSD is to make that parameter controllable. When β is high, the model stays close to the starting point and changes little. When β is low, it explores more and can improve more, but risks drifting too far from the original behavior. It’s the same type of balancing found in other policy optimization methods, but applied here to self-distillation.

The second technical contribution is the use of return-to-go credit assignment. In standard training, the reward arrives at the end of the sequence, and all tokens receive the same signal, even those at the beginning that may not matter much to the final result. Return-to-go recalculates the credit for each token based on the partial reward from that point onward, better aligning the update of the single token with its actual contribution to the sequence.

The numbers: on math reasoning benchmarks, β-OPSD improves average performance over vanilla OPSD across the full range of Qwen3 models tested (1.7B, 4B, 8B), with the most marked gain, +5.74 points on average, on the 1.7 billion parameter model. The paper also reports greater training stability, measured as lower variance in performance during the process.

The limitations. Results are on math reasoning tasks, not on general agentic tasks or other domains like code or logic. The models tested are Qwen3, a specific family: it’s not known whether the method transfers directly to other architectures. The paper doesn’t compare β-OPSD with more advanced RL techniques (like GRPO or PPO), only with vanilla OPSD. Code is available on the project page, but at publication time there are no ready-to-use scripts to replicate results on other datasets.

For those following the thread of distillation as a post-training technique, β-OPSD fits into the same space as SDR from AWS and ISO from UT Austin, where the theme was fine-tuning the model without destroying what it already knows. Here the angle is different: making a technique engineerable that promised much but was reliable only with heavy manual work.