MO KD

[junxnone]

知识蒸馏

• Knowledge Distillation 知识蒸馏
• Teacher/Student Model
  • Teacher Model 的输出 作为 Soft Target Training Student Model
  • Student 学习 Teacher 泛化能力
• 用途
  • 模型压缩(小模型学习大模型的泛化能力)
• History
• Distilling the Knowledge in a Neural Network

---

Trend

Reference

• 2020-6 Knowledge Distillation: A Survey [Paper]
• Awesome list
  • Awesome Knowledge Distillation
  • Awesome Knowledge-Distillation