Zhengxiong Luo

icon company BAAI icon location Received the BEng degree in SJTU in 2018. Received Ph.D. degree in CRIPAC@CASIA. Currently a full-time researcher in BAAI.

Results 11 comments of Zhengxiong Luo

[BUG] DeepSpeed allocates GPU memory in an unbalanced way.

The same problem. I am using zero3 to train a transformer with multi-nodes. On each node, deepspeed allocates much larger memory to the GPU with local_rank=0.