Technical limitations in processing arbitrary datasets?

[phinate]

Hi! I was interested in fine-tuning midi-ddsp on a set of midi files I already have in order to generate midi from that context (lmk if that's not possible and I've misunderstood), but I see that you don't currently support the processing of arbitrary dataset.

Was just wondering what you're hitting technically there? Or is it just the lack of a pipeline to process a random set of midi files using ddsp's data handling tools?

Would be happy to look into a PR if the scope is fairly well defined!

[lukewys]

Hi @phinate ! Are you planning to generate MIDI that is similar to your midi files or you are planning to generate audio renderings of MIDI files similar to your midi files? MIDI-DDSP is a model that takes in MIDI and generates audio rendering of the MIDI. If you want to generate MIDI, there are other awesome models that generate MIDI and can train on your own MIDI dataset. To name a few, you could look at CocoNet, MelodyRNN, Music Transformer, or Piano Transformer, all of above can be found in https://github.com/magenta/magenta.

MIDI-DDSP on the other hand generates audio and trains on audio recordings of monophonic instrument playing (as output) and the MIDI file that correspond to each recording (as input). By saying "don't support the processing of arbitrary dataset", we refer to currently MIDI-DDSP does not support train on any audio dataset with MIDI files.

Best, Yusong