Extending musicBERT for octuple generation and style based remix

[AniketRajpoot]

Hii! Recently, I have been working creating user friendly remixing tool with my friend @tripathiarpan20 known as midiformers. I am wanting to further extend the tool by :

• Firstly, I would be integrating genre and style based remixing which allows a user to remix the midi in particular style like jazz, pop, rock etc. I was thinking of prepending a class token similar to Jukebox (they prepend artist and genre embedding) would it be enough to condition the octuples and would fine-tuning on TOP-MAGD with class labels work ? I would be glad to know your thoughts on this.
• Secondly I would like to train a BERT-to-BERT encoder decoder model to perform autoregressive task again I if the above task succeeds, I will include genre conditioning. But I am not sure if this is possible with musicBERT. Can you help me through ?

Thank you so much for your amazing work. It is really inspiring for us!

[mlzeng]

Hi @AniketRajpoot

• I think prepending class tokens will work. But the genre classes in TOP-MAGD are general and ambiguous, so the condition may be too weak for generation tasks. To get better results, you may need to prepend artist tokens (like Jukebox) to make the condition stronger.
• You can run MusicBERT (mask some tokens of the part to be remixed and reconstruct them) multiple times to get better results. However, BERT-variant models are not for generation tasks. if you want SOTA generation results, you may need to train a new encoder-decoder model.

[AniketRajpoot]

Thank you so much for your reply @mlzeng!

• Thank you for your information regarding conditioning, I just wanted to ask one follow up question : How much training is required for this task ?, because I don't think I have much computational resources from my end.
• Thank you noted! Do you think Transformer-xl is good enough or do you think other architectures like T5 will outperform xl for generation task ?

Thank you so much again!