JEN-1-COMPOSER-pytorch(WIP)

Unofficial implementation JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generation(https://arxiv.org/abs/2310.19180)

README

📖 Quick Index

• 💻 Installation
• 🐍Usage
• 🧠TODO
• 🚀Demo
• 🙏Appreciation
• ⭐️Show Your Support
• 🙆Welcome Contributions

💻 Installation

git clone https://github.com/0417keito/JEN-1-pytorch.git
cd JEN-1-pytorch
pip install -r requirements.txt

🐍Usage

Sampling

import torch
from generation import Jen1

ckpt_path =  'your ckpt path'
jen1 = Jen1(ckpt_path)

prompt = 'a beautiful song'
samples = jen1.generate(prompt)

Training

torchrun train.py

Dataset format

Json format. the name of the Json file must be the same as the target music file.

{"prompt": "a beautiful song"}

How should the data_dir be created?

'''
dataset_dir
├── audios
|    ├── music1.wav
|    ├── music2.wav
|    .......
|    ├── music{n}.wav
|
├── metadata
|   ├── music1.json
|   ├── music2.json
|   ......
|   ├── music{n}.json
|
'''

About config

please see config.py and conditioner_config.py

🧠TODO

• [ ] Extension to JEN-1-Composer
• [ ] Extension to music generation with singing voice
• [ ] Adaptation of Consistency Model
• [ ] In the paper, Diffusion Autoencoder was used, but I did not have much computing resources, so I used Encodec instead. So, if I can afford it, I will implement Diffusion Autoencoder.

🚀Demo

coming soon !

🙏Appreciation

Dr Adam Fils - for support and brought this to my attention.

⭐️Show Your Support

If you find this repo interesting and useful, give us a ⭐️ on GitHub! It encourages us to keep improving the model and adding exciting features. Please inform us of any deficiencies by issue.

🙆Welcome Contributions

Contributions are always welcome.

Citations

@misc{2310.19180,
Author = {Yao Yao and Peike Li and Boyu Chen and Alex Wang},
Title = {JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generation},
Year = {2023},
Eprint = {arXiv:2310.19180},
}