[MusicBERT] Compatibility with latest fairseq repo

[tripathiarpan20]

Hi!

I have been thinking about experimenting with latest extensions/features of PyTorch and for that the MusicBERT code would need to be compatible with the latest version of fairseq repo (which is presumably compatible with latest PyTorch repo too).

Is it possible to make the changes in the code soon for compatibility with latest fairseq repo? (v 0.12.2) I would have tried to do that myself and raised a PR but currently I got my hands full.

Thanks!

[mlzeng]

Hi @tripathiarpan20

Migrating the code should be easy. But migrating the pre-trained checkpoints is tricky.

The MusicBERT plugin overrides a few methods of the original RoBERTa-related classes. If the original interfaces of methods are changed, then you may need to modify the code to adapt to the new interfaces.

[tripathiarpan20]

I see, I shall try doing these actually and update in a few days!.

[tripathiarpan20]

Basically what I have noticed (for the RobertaEncoder class) is that:

• In the fairseq v0.10.1 implementation, TransformerSentenceEncoder (inherited by OctupleEncoder) encapsulated both the sentence embeddings as well as the Transformer encoder layers.

• Meanwhile, the v0.12.2 implementation encapsulates sentence embedding into the default PyTorch torch.nn.Embedding class and the Transformer encoder layers into TransformerEncoder class.

Porting the checkpoints should also be possible after making changes to the current MusicBERT plugin.