candle
candle copied to clipboard
huggingface
AudioGen/MusicGen
Are there any plans to implement the AudioGen/MusicGen models from Facebook?
Actually I sketched a musicgen implementation a while back that is still in the repo here. At this point we didn't have the conv1d and lstm ops so I ended up not finishing it but now that these are implemented it would probably be a good idea to resurect these (the clip part of musicgen has already been merged in candle-transformers as it could be used outside of musicgen).
A bit OT but is candle-transformers meant to implement all pipelines from huggingface/transformers (like an extended version of rust-bert)?
I've been trying a few models so I could help there. Are PRs welcome?
The scope of candle-transformers is not totally settled yet. The workflow I would like to encourage for new models is:
- Start by developing the model in its own separate repo.
- If it works well, and the model is generic enough then we can consider moving it to
candle-transformers.
Tagging with help-wanted in case anyone wants to take a stab at this. There is some ongoing work in candle-examples but the forward passes are not implemented yet - you may also prefer starting from scratch.
Note that the encodec model is now available in candle so musicgen should be a pretty easy add at this point if anyone wants to give it a shot.
@LaurentMazare - I have implemented forward method for musicgen model. If it's ok I'd like to also implement generate method.
Great, yeah it would be pretty neat to also have the generate method in the example, that would make it far easier to test the forward method etc so feel free to do it in the PR that you've started minting!