l241025097

Results 3 comments of l241025097

[BUG]: fp32 param and grad have different shape torch.Size([5064704]) vs torch.Size([128000]) when use lora_rank=4 at stage 1

This error occurs when I use both --lora_rank and --grad_checkpoint. Either use --lora_rank or --grad_checkpoint.

[BUG] Fail to compile Deepspeed 0.9.0 with CUDA 11.7 and PyTorch 1.13.1 with Docker.

我发现，通过DS_BUILD_OPS=1参数去build的话会失败，于是我逐个op的去build。 DS_BUILD_CPU_ADAM=1 DS_BUILD_FUSED_ADAM=1 DS_BUILD_FUSED_LAMB=1 DS_BUILD_SPARSE_ATTN=1 DS_BUILD_UTILS=1 DS_BUILD_AIO=1 这些op加上都能build成功，一旦加上DS_BUILD_TRANSFORMER、DS_BUILD_TRANSFORMER_INFERENCE、DS_BUILD_STOCHASTIC_TRANSFORMER等XX_TRANSFORMER的op就会报build失败，具体为什么失败也看不出来。 另外，执行single_gpu时能正常运行，但是执行single_node时报错，如下： [2023-05-02 10:11:48,598] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.9.2+b0d9c4d0, git-hash=b0d9c4d0, git-branch=master [2023-05-02 10:11:48,599] [INFO] [comm.py:616:init_distributed] Distributed backend already initialized [2023-05-02 10:11:49,907]...

[BUG] Fail to compile Deepspeed 0.9.0 with CUDA 11.7 and PyTorch 1.13.1 with Docker.

> 我发现，通过DS_BUILD_OPS=1参数去build的话会失败，于是我逐个op的去build。 DS_BUILD_CPU_ADAM=1 DS_BUILD_FUSED_ADAM=1 DS_BUILD_FUSED_LAMB=1 DS_BUILD_SPARSE_ATTN=1 DS_BUILD_UTILS=1 DS_BUILD_AIO=1 这些op加上都能build成功，一旦加上DS_BUILD_TRANSFORMER、DS_BUILD_TRANSFORMER_INFERENCE、DS_BUILD_STOCHASTIC_TRANSFORMER等XX_TRANSFORMER的op就会报build失败，具体为什么失败也看不出来。 > > 另外，执行single_gpu时能正常运行，但是执行single_node时报错，如下： [2023-05-02 10:11:48,598] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.9.2+b0d9c4d0, git-hash=b0d9c4d0, git-branch=master [2023-05-02 10:11:48,599] [INFO] [comm.py:616:init_distributed] Distributed backend already...