LAVIS

LAVIS copied to clipboard

AISuperMa

How to use your own dataset to train and fine-tune the VQA task of BLIP2-flant5xl

24

Hi, thank you very much for open source. I want to use my own Image and caption, and QA data to fine-tune the BLIP2 data. Should my process be to...

xcxhy

The Problem on blip2_feature_extraction Example

1

Hello author, in the example/blip2_feature_ectraction.ipynb file, it seems that is not no blip2_feature_extractor model file mentioned in your code. I can't find the model in the model zoo, should I...

xcxhy

Hi, `torch.amp.autocast` is used in the [BLIP2-T5 model](https://github.com/salesforce/LAVIS/blob/7aa83e93003dade66f7f7eaba253b10c459b012d/lavis/models/blip2_models/blip2_t5.py#L218), while this API is added in [PyTorch 1.12.0+](https://discuss.pytorch.org/t/module-torch-has-no-attribute-amp/172026/2). An updated requirement for the torch version may be needed, not only to solve...

yqy2001

Resume training from checkpoint

1

When I try to feed in a `checkpoint_best.pth` from a previous run to resume training from checkpoint, I get an error at this line (https://github.com/salesforce/LAVIS/blob/main/lavis/runners/runner_base.py#L628) on missing keys in state_dict....

kttian

BLIP2 models would stop working with next release of tranformers (4.27.0)

3

I've been trying to use `transformers` to convert BLIP2 models into compatible format (https://github.com/huggingface/transformers/blob/main/src/transformers/models/blip_2/convert_blip_2_original_to_pytorch.py) and ran into some issues, specifically this commit https://github.com/huggingface/transformers/pull/21405 breaks compatibility between `lavis` and `transformers`. So...

AstraliteHeart

Hi, Thanks for your job. Can you provide the download link for the synthetic data of BLIP-2? How is this data used? Is there a code for it?

mxin262

Pre-Training BLIP2 Log

8

Could you release the log of pretraining blip2 of stages 1 and 2? When I tried to retrain the model, the loss seemed difficult to reduce.

phellonchen

How to handle multiple images with Blip2 models ?

1

How to handle multiple images with Blip2 models? I have a large number of questions which require more than one image to answer for VQA task, like 1 questions vs...

rr191211

BLIP2 Cuda out of memory issue

6

Hi, Thanks a lot for your work on BLIP2! I am using it to fine-tune my own dataset (similar format to COCO) on a 3090 GPU with 24GB memory. However,...

shengyi4