Researchers at the University of Southern California propose TOPL (Token-Level Off-Policy Labeling), a post-training method that changes the granularity of feedback. Instead of scoring the model’s entire response, TOPL marks individual tokens as good or bad and trains the model to recognize them. The model learns to avoid wrong tokens, not to maximize a global score that can be gamed.

This matters to you because reward hacking, the phenomenon where a model learns to score well without actually improving, is one of the most annoying practical problems when fine-tuning a model for repetitive tasks. If you’ve ever tried to correct an assistant that “stops” making mistakes during training only to fall back into the same error in production, you know what we’re talking about. Token-level feedback makes the correction signal finer: the model knows where it went wrong, not just that it went wrong.

The pattern is the same we’re seeing across tools: as we reported yesterday on textual feedback in post-training, research is moving away from blunt scoring toward more granular and interpretable signals. TOPL pushes granularity down to the individual token, and the authors show that LoRA adapters learned this way function like steering vectors, vectors that shift model behavior in readable ways.

Tests cover 11 summarization datasets and extend to machine translation, with out-of-distribution generalization. The stated limitation: it’s demonstrated on faithful generation tasks (generation faithful to input data), not on free-form reasoning or coding. No public code yet.

In detail

What came before.

Standard post-training works like this: the model generates a response, a judge (human or another model) scores the entire sequence, and the model updates its weights to produce higher-scoring responses. This approach, called outcome-level or sequence-level, has a known flaw: the model learns to game the score. It finds patterns that satisfy the judge without improving real quality. A concrete example: in a summarization task, the model can learn to produce shorter summaries (because the judge penalizes length) without becoming more faithful to the original document.

Token-level methods already exist, but they operate on-policy: they generate tokens with the model in real time and correct them one by one. It’s expensive and unstable, because you’re training the model on its own outputs while producing them.

What changes with TOPL.

TOPL is off-policy: it takes already-generated responses (from any model), marks tokens as good or bad, and trains the model to classify which token is correct and which is not. The key shift is that the model doesn’t learn to directly generate good tokens, but to recognize them. This avoids distribution shift (the model isn’t pushed to produce tokens it doesn’t yet know how to generate well) and reduces reward hacking, because the signal is local and specific.

The authors explain it with an analogy that works: instead of telling a writer “this article scores 6”, you tell them “this word is imprecise, this sentence is redundant, this citation is correct”. Targeted feedback corrects better than a global score.

The numbers.

On 11 summarization datasets, TOPL beats both sequence-level and on-policy token-level baselines, with out-of-distribution generalization. It transfers to machine translation without retraining. Ablations confirm that the token-level signal is critical: the sequence-level analog of TOPL doesn’t achieve the same results. Finally, LoRA adapters learned with TOPL function as steering vectors: vectors that, applied at inference, shift model behavior in interpretable directions (more faithfulness, fewer hallucinations).

Limitations.

The tested tasks are faithful generation: summarization and translation, where an objective notion of “correct token” exists (the token does or doesn’t match the input data). On tasks where correctness is less binary, like free-form reasoning or coding, it’s unclear how to label individual tokens as good or bad. No public code yet, so no independent reproduction. The paper’s an early upvote on HF Daily Papers: an early signal, not consolidated confirmation.