Top Highlights
-
The challenge: Continual learning lets models acquire new skills from ongoing data, but ordinary fine-tuning can cause catastrophic forgetting of old abilities.
-
How OPSD works: It combines on-policy training—learning from the model’s own outputs—with self-distillation: an EMA-updated copy of the model acts as a teacher, using expert demonstrations to guide the student.
-
Why it’s promising: Early studies report less forgetting than standard fine-tuning, stronger performance on new tasks, and better training-token efficiency than GRPO in some settings.
-
Important caveats: Some OPSD variants reduce the variety of a model’s answers, and early tests found performance drops in reasoning-focused “thinking” models—so it’s promising, but not a universal fix.
Could On-Policy Self-Distillation Revolutionize Post-Training?
A gentler way to teach AI
Post-training gives a foundation model new skills. But it can also make the model forget old ones, a problem known as catastrophic forgetting. That risk matters when teams update models for new fields, languages or tasks.
On-policy self-distillation offers a promising approach. The model generates its own answers, while a matching “teacher” version sees expert examples. The teacher then guides the model toward better responses. Because both versions start from the same model, the guidance may help it learn without pulling it too far from its existing abilities.
Early results show promise
So far, studies report encouraging results. In one set of experiments, a model learned several distinct tasks while retaining earlier gains. Other tests found that the method could outperform supervised fine-tuning and use fewer training tokens than a reinforcement-learning comparison.
That could make updates more practical. Instead of rebuilding a model from scratch, organizations may be able to add skills while preserving much of what the model already does well. Still, these findings come from a young and fast-moving research area. Strong benchmark results do not guarantee similar gains in everyday use.
Adoption needs more evidence
The method also has trade-offs. Some versions reduce the variety of a model’s answers. That matters when a system needs to explore several approaches, such as when solving a hard problem or choosing among possible actions. Early tests also suggest that the method can hurt some reasoning-focused models.
So, OPSD looks less like a finished solution and more like a useful new tool. Researchers need to test it across more models, tasks and real-world settings. If they can preserve answer diversity and adapt the method for reasoning models, OPSD could become a valuable part of post-training—not a replacement for every other approach.
Stay Ahead with the Latest Tech Trends
Learn how the Internet of Things (IoT) is transforming everyday life.
Stay inspired by the vast knowledge available on Wikipedia.
AITechV1
