LLaMa wont load when using deepspeed

[catalpaaa]

Describe the bug

LLaMa wont load when using deepspeed, it will just get stucked on

Loading llama-7b...

and taking up all the ram before freezing the system.

Is there an existing issue for this?

• [X] I have searched the existing issues

Reproduction

fellow the tutorial for enable deepspeed, using llama 7b hf.

Screenshot

No response

Logs

only log shown is Loading llama-7b...

System Info

3080ti 12gb on ubuntu 22

[catalpaaa]

btw theres 64gb swap but looks like its not used.

[catalpaaa]

Also it looks like whenever the model is splite into mutilple bins then stuff just freeze. To me, deepspeed just doesnt wanna use the swap, it takes up all the buff/cache then dose nothing.

[cyrcule]

Also it looks like whenever the model is splite into mutilple bins then stuff just freeze. To me, deepspeed just doesnt wanna use the swap, it takes up all the buff/cache then dose nothing.

Have you found a way to force deepspeed to use swap?