unilm Any plans to release a sample code for MINI-LM distillation?

Any plans to release a sample code for MINI-LM distillation?

Open leimao opened this issue 5 years ago • 14 comments

Hi,

Thank you for releasing the distilled MINI-LM models from pre-trained Transformer models. I wonder if you have any plans to release a sample code for MINI-LM distillation implementations in either TensorFlow or PyTorch? If people have different pre-trained Transformer models and they want to distill the knowledge, they might be benefited from such scripts. Thank you very much.

Best,

Lei

Apr 06 '20 22:04 leimao

@WenhuiWang0824

Apr 07 '20 15:04 donglixp

Hi, I was just wondering if there is any update regarding this? Having these scripts would be amazingly helpful! 🙂

Thank you so much!

Aug 04 '20 22:08 VenkatKS

Do you have any update on this please ?

Mar 05 '21 10:03 padipadou

@WenhuiWang0824

Bump on this thread. Any update on this would be appreciated. I think there are many researchers and the community at large that would benefit from being able to distill non-BERT LMs.