A Microsoft Research paper introduces “Experiential Learning” (EL), a post-training method that decouples reinforcement learning from textual coaching. Instead of reducing a judge model’s evaluation to a single number, textual feedback is preserved and internalized by the model as experiential knowledge.

Post-training on open-ended tasks (writing, answering questions without verifiable answers, helping with creative work) today compresses evaluation into a scalar reward. Two responses of different quality can receive the same score, and the model learns to chase the number instead of truly improving. EL addresses this limitation: the judge stops giving a score and writes an evaluation that the model can read, understand, and absorb. The paper’s results show better generalization outside the training set and less reward hacking.

As we covered on July 14, the trend in post-training is toward modularity: separating tasks (exploration, alignment, and now coaching) makes each phase more controllable and reusable. This paper pushes the same idea forward, on the feedback signal side.

In Detail

The problem the paper addresses is as old as RLHF itself: when training a model with reinforcement learning on open-ended tasks, you need a numeric signal to tell the algorithm whether a response is good or not. To get it, you use a model (the “judge”) that evaluates responses according to a rubric, a list of criteria. The judge reads the response, evaluates each criterion, and then everything gets compressed into a number. If the response has excellent structure but weak arguments, the final number might be 0.7. If another response has mediocre structure but strong arguments, it might also be 0.7. The model learning from that number can’t distinguish the two profiles.

EL (Experiential Learning) changes the channel. The judge writes a detailed textual evaluation, like a human reviewer with a red pen. This evaluation gets “distilled” into experiential knowledge: it conditions a teacher model that generates better responses, and the main model internalizes this knowledge in its weights through a process called “on-policy context distillation.”

The difference from classical RL is signal bandwidth. A number is a narrow channel: it only tells you “better” or “worse.” Text tells you what’s better, why, and what should change. The paper shows that this wider channel produces denser supervision and preserves fine-grained preferences between high-quality responses.

The results, measured on two model families with feedback from the model itself or from a proprietary model, show three things. First: EL beats RL based on rubric on held-out and unseen open-ended tasks. Second: it generalizes better outside the training distribution. Third: it mitigates reward hacking, the phenomenon where the model learns to fool the judge for a high score without actually improving.

For those experimenting with post-training, the practical lesson is that feedback format matters as much as its quality. Structured textual feedback costs more tokens and more compute, but produces models that understand what makes them better instead of chasing a proxy metric. It’s the same direction as the modular post-training framework from July: separating phases makes each piece more controllable.

The limitations. The paper is from Microsoft Research and has 7 upvotes on Hugging Face at publication time: it’s a fresh result, with limited community review. Complete details on architecture and hyperparameters will be in the full paper and code, which the authors announce at aka.ms/el-code but isn’t yet verified as accessible. Evaluations cover open-ended tasks (non-verifiable): the method applies where reward can’t be computed deterministically, and doesn’t replace RL on tasks with exact answers (math, coding). As with any post-training paper, reproducibility depends on access to compute resources for training, which isn’t available to everyone.