A paper from the Qwen team (CASIA) builds a controlled environment to study an open question: how does a foundation model learn multi-turn planning over long horizons. The answer unfolds across three phases, from acquisition during pre-training to integration via multi-teacher on-policy distillation.

The key results matter for agent trainers. During pre-training, building an explicit world model through chain-of-thought reasoning helps generalization, while atomic abilities alone cannot compose complex plans. Suboptimal trajectories cause severe damage: errors amplify at each turn. In post-training, on-policy distillation (OPD) has a wider effective region than GRPO when data is scarce and the horizon extends, because it provides more consistent update directions. Multi-teacher OPD (MOPD) integrates capabilities from multiple environments by converging toward shared patterns, but if patterns conflict, interference is severe.

For practitioners doing self-play on agents, the operational message is direct. Trajectory quality matters more than quantity: an agent learning from its own rollouts over long horizons needs correct paths, because an early error doesn’t dissolve, it multiplies. And when distilling from multiple teachers, the compatibility of their planning styles determines whether competencies add up or cancel out.

It’s the same thread as the Experience Distillation we discussed on July 26: the agent learns from its own rollouts, not just from external demonstrations.

In detail

The paper’s starting point is a paradox in agent research. Foundation models are trained on billions of web pages, but no one can precisely say what in those data teaches them to plan over multiple turns. The model works, but the training ingredients that produced the observed planning capability remain unknown. The CASIA team built a controlled multi-turn environment, where every variable is manipulable, and used it to study planning across three distinct phases.

Phase 1: acquisition in pre-training. Here, the ingredients of pre-training that make a model capable of planning are isolated, before any fine-tuning. The main finding is that building an explicit world model, through chain-of-thought reasoning that models state transitions, produces better generalization over long horizons. Atomic abilities, taken individually, don’t compose well into complex plans: some data on long horizons is needed from pre-training itself. And suboptimal trajectories are particularly harmful, because in a ten-step plan, an error at step three contaminates everything after. Reinforcement learning practitioners know this intuitively, but the paper quantifies it in a controlled setting.

Phase 2: shaping in post-training. The paper compares two techniques: GRPO (Group Relative Policy Optimization, a reinforcement learning method that evaluates groups of responses relatively) and OPD (On-Policy Distillation, where the model learns from its own on-policy generated trajectories, not from static demonstrations). Using mutual information, the authors distinguish two components of planning: general patterns (how to structure a plan) and task-specific knowledge (which steps to take in a particular domain). For general patterns, they identify three regions where post-training is unnecessary (the model already knows it), effective, or unsupported. OPD has a wider effective region than GRPO in low-quality and long-horizon settings, because it provides more consistent update directions. For task-specific knowledge, distilling procedures never seen from a teacher with different knowledge can damage the student’s pre-existing world model without stabilizing the new knowledge.

Phase 3: integration with MOPD. Multi-teacher on-policy distillation means taking multiple teachers, each expert in a different environment, and converging the model toward shared planning patterns. Compatible patterns enable cross-environment generalization. Partially shared patterns support continual learning. Completely conflicting patterns cause severe interference: if two teachers plan in incompatible ways, the student doesn’t learn both, it gets worse.

The limitations. The study happens in a controlled environment, which is both its strength (variable isolation) and its main limitation. Transferability to the real world, with noisy data and non-simulable environments, remains to be shown. The authors don’t report benchmark numbers comparable to production models in their abstract. Code and dataset are published on GitHub and Hugging Face for those who want to replicate.

For those building agents in production, the most direct practical implication concerns the post-training phase. If you’re fine-tuning an agent on multi-turn tasks, the quality of training trajectories becomes the dominant variable. A single wrong step in a long sequence is worth more than ten wrong steps in short sequences, because the error propagates. And if you’re distilling from multiple teacher models, first verify that their planning styles are compatible.