BLIP2 Cuda out of memory issue

[shengyi4]

Hi, Thanks a lot for your work on BLIP2! I am using it to fine-tune my own dataset (similar format to COCO) on a 3090 GPU with 24GB memory. However, it shows Cuda out-of-memory error even with setting batch size as 1. After setting freeze_vit to True, the Cuda out-of-memory error goes away. Is it normal that have this error when using BLIP2? It worked ok with BLIP (BLIP1). Is it ok to fine-tune the model by freezing the vit? Please find my lavis/projects/blip2/train/caption_coco_ft.yaml file below. I am looking forward to hearing back from you. Thanks! model: arch: blip2_opt model_type: caption_coco_opt2.7b load_finetuned: False use_grad_checkpoint: True freeze_vit: True

datasets: coco_caption: # name of the dataset builder vis_processor: train: name: "blip_image_train" image_size: 364 eval: name: "blip_image_eval" image_size: 364 text_processor: train: name: "blip_caption" prompt: "a photo of " eval: name: "blip_caption"

run: task: captioning lr_sched: "linear_warmup_cosine_lr" init_lr: 1e-5 min_lr: 0 warmup_lr: 1e-8 warmup_steps: 1000 weight_decay: 0.05 max_epoch: 5 batch_size_train: 4 batch_size_eval: 4 num_workers: 4 accum_grad_iters: 1

max_len: 20 min_len: 1 num_beams: 3

seed: 42 output_dir: "output/BLIP2/Caption_coco"

amp: True resume_ckpt_path: null

evaluate: False train_splits: ["train"] valid_splits: ["val"] test_splits: ["test"]

device: "cuda" world_size: 1 dist_url: "env://" distributed: False