Finetuning ImageBind with LoRA

[fabawi]

I created a simple ImageBind finetuning example using LoRA: https://github.com/fabawi/ImageBind-LoRA

Make sure you clone it recursively to include the example dataset: git clone --recurse-submodules -j8 [email protected]:fabawi/ImageBind-LoRA.git

Install the requirements following the instructions provided in this repo, and run train.py

This should log your checkpoints, as well as separate LoRA if you'd like to update the original model without saving all the model params. More examples and finer control to be added soon

[WilTay1]

Can I know how you run the model to generate images?

[ChloeL19]

^^^ upvoting WilTay1's question. also wondering if you know how to train ImageBind without using LoRA?

[fabawi]

^^^ upvoting WilTay1's question. also wondering if you know how to train ImageBind without using LoRA?

@ChloeL19 you can train the model without LoRA using ImageBind-LoRA. Simply remove the --lora argument when calling train.py and set --full_model_checkpointing. I don't have the resources to fine-tune it but it should work in theory. Try it out on the toy dataset provided in the repo, and later you can try to implement dataloaders for the original datasets

[ChloeL19]

Great, thank you so much! I'll keep you posted depending on how far I get.

[chavinlo]

Can I know how you run the model to generate images?

late, but there are many repositories that just replace CLIP for ImageBind A good example is https://github.com/Birch-san/imagebind-guided-diffusion

[K-M-Ibrahim-Khalilullah]

Nice work. Thanks Could you a little explain why you used LoRA. Did the paper use it?

[llziss4ai]

There is a repo that uses LoRA to bind modalities to language.