try to fix Zero3 Memory Leak following @tohtana idea

[dumpmemory]

here i am following @tohtana 's modification from https://github.com/microsoft/DeepSpeed/issues/3002 to fix https://github.com/huggingface/peft/issues/161 . it worked with deepspeed 0.9.1 and torch 2.0. Thanks for @tohtana 's help.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[aashay96]

When will this be deployed?

[dumpmemory]

Hello @dumpmemory, great work getting this issue solved from DeepSpeed and raising the fix here. Could you apply the fix to all places in lora and adalora wherein F.linear is being used . That would solve the issue in all places

cool i will. Thanks

[dumpmemory]

@pacman100 pls help me to check it. i have made all F.linear replaced.

[dumpmemory]

Thank you @dumpmemory for iterating, LGTM! 🤗

Could you run make style and make quality to fix the quality issues?

yes, i will. I will also test new commits from deepspeed sides. Thanks again.

[pacman100]

Hello, @dumpmemory, there are still some code quality issues. Please resolve them to go ahead with the PR

[pacman100]

Hello, is this PR still required? As the DeepSpeed team fixed it in their codebase

[dumpmemory]

this pr is no longer required.