A paper featured on HuggingFace Daily Papers introduces Experience Distillation: a technique to encode into model weights what an agent learns from its interaction history, without running additional environment steps.

The problem is familiar to anyone building agents for repetitive tasks. An agent improves when you keep previous attempts in context, but those gains vanish as soon as you clear the window. Traditional fine-tuning on collected data recovers only 3.8% of that gain (paper on HuggingFace, July 22, 2026), because the model learns to imitate actions without grasping their underlying logic.

Experience Distillation retains at least 64.8% of in-context learning’s advantage across 749 software engineering tasks and six text-based games, and matches reinforcement learning baselines with 9.6x fewer environment interactions. This touches the same theme we discussed when covering OpenForgeRL: how to make agent learning efficient without prohibitive sampling costs. Here the sampling is already done, what remains is distilling the collected material.

For teams running agents in production that accumulate interaction traces, the direction is clear: agents that genuinely improve over time, not just session by session.

In Detail

In-context learning is the simplest mechanism to teach an agent: you put its interaction history in the prompt, what it tried, what worked, what didn’t, and the model uses that history to behave better. It works well because the model sees the entire experience and can reason about it. The catch is that context windows are workbenches, not archives. As history grows, it fills the window, and removing it sends the model back to square one.

The obvious solution is fine-tuning: collect interactions and train the model on that data. The paper tries this and the result disappoints. On software engineering tasks and text games, supervised fine-tuning recovers only 3.8% of what in-context learning had gained. The model learns to reproduce actions but loses the connection between environment observation and the decision that motivated them.

Experience Distillation takes a different path. Instead of training the model to replicate actions, it uses context distillation, a known technique that transfers information from context into model weights. Here it applies to complete interaction histories, not individual instructions or examples. The key point: no additional environment interaction is needed. You collect experience once, distill it, and get a model that has internalized part of what it learned in context.

The numbers, measured across 749 curated software engineering tasks and six text games, show that distillation retains at least 64.8% of in-context learning’s gain in both domains. Compared to classical reinforcement learning baselines, in-context learning followed by distillation matches their performance with at least 9.6x fewer environment samples.

What this means for builders. If you have an agent running repetitive tasks, debugging, search, data analysis, each session produces interaction traces that normally disappear. This paper suggests those traces, if distilled properly, can become permanent model improvement. They don’t replace reinforcement learning from scratch, but drastically reduce the environment interactions needed to reach the same result.

The limitations. The paper measures two specific domains; whether that 64.8% generalizes to different tasks is unclear. The gain is “at least” 64.8%, but variation across simple and complex tasks isn’t specified. Distillation requires access to model weights, so it doesn’t apply to teams using models via API without fine-tuning access. And the RL baseline comparison uses performance metrics, not robustness: a distilled model might be more fragile on out-of-distribution inputs than one trained with RL from the start.