Distill What You Want: A Unified View of On-Policy Distillation
Distillation is not only about choosing a loss—it begins with deciding what the student should learn. This post uses the centered log-ratio (CLR) transformation to turn target construction, weak-to-strong transfer, and multi-teacher distillation into an intuitive problem of vector composition and matching.