Zhewei Yao

Thanks a lot @ftian1. Look forward to the PR :)

That sounds good to me :)

Closed for now. Please re-open it if you need further assistance

Thank you @puyuanOT :). Yes, the LoRA replacement is based on the model arch (or the name)

How large memory does 4090 have? We have basic instruction in the tutorial/readme of step 3

Looks like the memory is enough. Can you share the training script you are using for now?

Close as no followup

@blldd is 15GB only for deepspeed initialization or it is the peak memory consumption during training? During training, a lot of memory is consumed by activation/compute and others.

During training, you will need a lot of memory for intermediate activation. Also, please take a look at this https://github.com/microsoft/DeepSpeedExamples/issues/299

Close the issue since there is no followup. Please reopen it if you still need more clarification.