How to run InstructBLIP

[ouhenio]

Hello!

I'm trying to run Vicuna InstructBLIP, but sadly, I can't make it work.

I installed LAVIS directly from your repo following the step 3 of the installation guide, and I'm using the following code:

import torch
from lavis.models import load_model_and_preprocess

device = torch.device("cuda") if torch.cuda.is_available() else "cpu"

model, vis_processors, _ = load_model_and_preprocess(
    name="blip2_vicuna_instruct", 
    model_type="vicuna7b", 
    is_eval=True, 
    device=device
)

With which I get the following error:

File "~/.../LAVIS/lavis/models/blip2_models/blip2_vicuna_instruct.py", line 82, in __init__
    self.llm_tokenizer = LlamaTokenizer.from_pretrained(llm_model, use_fast=False, truncation_side="left")

...

huggingface_hub.utils._validators.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': './llm/vicuna-7b'. Use `repo_type` argument if needed.

I guess this happens because inside the blip2/blip2_instruct_vicuna7b.yaml configuration llm_model is set to "./llm/vicuna-7b".

I tried to fix this by manually downloading the model from the pretrained value present in said yaml, and create a ./llm/vicuna-7b folder to store it but it still fails.

Any ideas on how can I run this?