Clipora - Low Rank Adapter Fine Tuning of OpenCLIP

Main Features

• Fine tune OpenCLIP models via LoRA

TODOS

• [x] Adapter saving
• [ ] Merging adapters with the original model
• [ ] Inferencing with the adapter
• [ ] Add better documentation
• [ ] Support for more logging

References

The links below have most of the information/implmentation here in much cleaner ways, all credit to them, I just frankensteined them together to make this.

• https://github.com/cloneofsimo/lora
• https://arxiv.org/abs/2106.09685
• https://github.com/facebookresearch/ov-seg
• https://huggingface.co/blog/lora
• https://github.com/huggingface/peft
• https://github.com/KyanChen/MakeMultiHeadNaive