Zhewei Yao
Zhewei Yao
Thanks a lot @ftian1. Look forward to the PR :)
That sounds good to me :)
Closed for now. Please re-open it if you need further assistance
Thank you @puyuanOT :). Yes, the LoRA replacement is based on the model arch (or the name)
How large memory does 4090 have? We have basic instruction in the tutorial/readme of step 3
Looks like the memory is enough. Can you share the training script you are using for now?
Close as no followup
@blldd is 15GB only for deepspeed initialization or it is the peak memory consumption during training? During training, a lot of memory is consumed by activation/compute and others.
During training, you will need a lot of memory for intermediate activation. Also, please take a look at this https://github.com/microsoft/DeepSpeedExamples/issues/299
Close the issue since there is no followup. Please reopen it if you still need more clarification.