litgpt icon indicating copy to clipboard operation
litgpt copied to clipboard

Published 20 hours ago • verified publisher icon Lightning-AI

support GaLore

Open batman-do opened this issue 11 months ago • 3 comments

Can repo support GaLore soon ? https://github.com/jiaweizzhao/GaLore

batman-do avatar Mar 09 '24 04:03 batman-do

Hello @batman-do

It looks like GaLore is a drop-in replacement for the PyTorch optimizer, meaning that you can take any script that does pretraining/fine-tuning and replacement the part that defines optimizer.

Andrei-Aksionov avatar Mar 09 '24 10:03 Andrei-Aksionov

Hello @batman-do

It looks like GaLore is a drop-in replacement for the PyTorch optimizer, meaning that you can take any script that does pretraining/fine-tuning and replacement the part that defines optimizer.

Can u support fsdp + qlora soon , i stuck oom when max_seq_length large when fine-tune lora > 24gb with model 7b

batman-do avatar Mar 09 '24 13:03 batman-do

Can repo support GaLore soon ?

In the works via #1192 !

rasbt avatar Mar 25 '24 21:03 rasbt