How to use single machine multi card reasoning

[hcxss]

How to use single machine multi card reasoning

[zinengtang]

Is your GPU memory below 30GB ? If not, 32GB mem or 40GB mem is enough

[hcxss]

单卡24GB

---Original--- From: "Zineng @.> Date: Fri, Jun 30, 2023 16:50 PM To: @.>; Cc: "Changxiu @.@.>; Subject: Re: [microsoft/i-Code] How to use single machine multi card reasoning(Issue #66)

Is your GPU memory below 30GB ? If not, 32GB mem or 40GB mem is enough

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[zinengtang]

Let me try to make a PR to incorporate fp16 and it should be less than 24GB mem.

[hcxss]

Can't configure single machine multi card

---Original--- From: "Zineng @.> Date: Fri, Jun 30, 2023 17:04 PM To: @.>; Cc: "Changxiu @.@.>; Subject: Re: [microsoft/i-Code] How to use single machine multi card reasoning(Issue #66)

Let me try to make a PR to incorporate fp16 and it should be less than 24GB mem.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[zinengtang]

what's the reason of it?

[zinengtang]

you can try deepspeed stage 3 for model parameter states partitioning

[PHOENIXFURY007]

Can you elaborate on this a little more? Thanks

[zinengtang]

so in using deepspeed stage 3, the model parameters states will be split among gpus and thus reducing the model memory. deepspeed is supported in pytorch, pytorch lightning, etc.