BLIP2 fine-tuning on custom LLM+dataset

[visheratin]

Hi!

I want to extend BLIP2 capabilities to another language. I have a pre-trained LLM (T5 family) and a dataset with image captions. Could you please help me understand my next steps to train the model? Do I need to perform pre-training, or can I use pre-trained ViT along with my T5 model and do fine-tuning?

[joaopedrosdmm]

Hey,

Looking to do a similar project. Did you ever found an answer to your question? Would love to get some clarification.

[shams2023]

嘿，

想做一个类似的项目。您找到问题的答案了吗？希望得到一些澄清。

Hello! I am also working on fine-tuning the image capture task using my own dataset, BLIP2. Have you succeeded

[simeneide]

Working on the same, did anyone have any luck?

[betterftr]

any progress?

[shams2023]

Can you share code?