Is there any way to distill translation models?

[AIikai]

Prepare to compress the translation model trained by fairseq, such as distillation and pruning. Can you give me some advice? Thank U.

[robotsp]

@AIikai I have a similar question. Do you have any ideas?

[VarunGumma]

@AIikai @robotsp I am also looking to distill and prune a few LLMs. Any leads?

[HeegonJin]

@AIikai @robotsp @VarunGumma I am also trying to implement KD with the modification of fairseq

[VarunGumma]

@robotsp @AIikai @HeegonJin please redirect here for KD in fairseq