LAVIS

LAVIS copied to clipboard

[BLIP2] How to perform stage 1 Vision-Language Representation bootstraping

1

I think that stage 1 learning, that means visual-language representation learning with those three objectives mentioned in the article is not yet implemented. Am I right? Not implemented `load_pretrained: False`...

klima7

Unable to train with "clip_L" VIT option (or other generic VIT)

1

I tried to train with clip_L vision encoder, (by adding vit_model: "clip_L" to model train config), but it seems the QFormer checkpoint loaded by default at this line (https://github.com/salesforce/LAVIS/blob/main/lavis/models/base_model.py#L100) is...

kttian

transformers 4.27 compatability

5

I have to use transformers 4.27 because latest version of clip-interrogator requires that specific version. After upgrading transformers from 4.26 to 4.27, I had this issue. ``` ╭─────────────────────────────── Traceback (most...

gunesevitan

[BLIP2]: Low accuracy of zeroshot VQA of BLIP2-opt-2.7b

12

Hi, Thank you for your great work BLIP2. I find there is no zeroshot VQA evaluation code for BLIP2-OPT, so I create one, refering to the code of FLAN-T5. However,...

YuanLiuuuuuu

fix on blip_caption.py: prepare prompt for beam search, aovid tensor …

1

…mismatch in later XMLBert decoder. fix issue #241

Luodian

Can BLIP2 work with small batch size?

3

Can BLIP2 work with small batch size like 64,128,256?

yxchng

Is it possible to do this without going from GPU -> CPU -> GPU? Code: ```py vis_images = [self.vis_processors["eval"](Image.fromarray(image.cpu().numpy())).unsqueeze(0).to('cuda').squeeze(0) for image in images] features_image = self.blip2_model.extract_features({"image": torch.stack(vis_images)}, mode="image") ``` extract_features...

Suhail

use_dist_eval_sampler arguments in retrieval evaluation

1

When I set use_dist_eval_sampler=True in retrieval evaluation, the result is very bad. But when I set use_dist_eval_sampler False, the result is very good

vanpersie32

How to use BLIP2 finetuned model

2

I train a finetune model use command: `python train.py --cfg-path lavis/projects/blip2/train/pretrain_stage2.yaml` my env is  but when i use finetuned model to generate caption, the error happend `RuntimeError: Sizes of...

luozhiping

AttributeError: 'NoneType' object has no attribute '__getstate__'

2

I tried to run the code in huggingface (https://huggingface.co/Salesforce/blip2-opt-2.7b ) ```python import requests from PIL import Image from transformers import BlipProcessor, Blip2ForConditionalGeneration processor = BlipProcessor.from_pretrained("Salesforce/blip2-opt-2.7b") ``` However, I meet this...

fly-dragon211