Any idea on how to reproduce vicuna in Nvidia L4?

[Ejafa]

Hi,, I have been trying to reproduce vicuna in 4xL4 which is 4*24=96 gb vram. What approach should I follow?

Train.py oom error Train_mem.py oom error Left is Lora but couldn't find deepspeed config which originally follows this repo. Can you guys suggest me any other alternative? Flan T5 looks promising but inference is not added yet. Similar one I have seen as far is Flan-alpaca did fine-tuning on sharegpt dataset with Lora.

Any suggestions on this will be gladly welcome for my research. Thanks

[zhisbug]

refer to this reply: https://github.com/lm-sys/FastChat/issues/543#issuecomment-1520909606

it is unlikely you can fine-tune any version of vicuna with only 96gb total VRAM.

[surak]

@Ejafa that looks like lack of hardware, not much one can do here. Mind if we close this one?

[infwinston]

@Ejafa feel free to open if you have any other question.