Hello everyone, can we perform Lora or other fine-tuning based on the Vicuna model? How much graphics memory is required?

[qimingyangyang]

Hello everyone, can we perform Lora or other fine-tuning based on the Vicuna model? How much graphics memory is required?

[jcao-ai]

As for 7B model, LoRA consumes about 17GB gpu memory.

[mvuthegoat]

As for 7B model, LoRA consumes about 17GB gpu memory.

If it consumes only 17GB, why did I get this message? OutOfMemoryError: CUDA out of memory. Tried to allocate 64.00 MiB (GPU 5; 23.70 GiB total capacity; 23.29 GiB already allocated; 58.56 MiB free; 23.29 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[jcao-ai]

Did you enable gradient_checkpointing ?