DeepSpeed
DeepSpeed copied to clipboard
[BUG] Load fused_adam_cuda failed
Describe the bug
I install the deepspeed with DS_BUILD_OPS=1 pip install deepspeed
and compile the extra keenal, but I get err when I use zero optimizer, I get fused_adam_cuda = FusedAdamBuilder().load()
, TypeError: 'NoneType' object is not callable
Expected behavior A clear and concise description of what you expected to happen.
ds_report output
I run ds_report
and get this error
System info (please complete the following information):
- OS: CentOS
- GPU count and types: 1machines with x8 A100s each
- Python version: 3.9.12
- Cuda:11.3
- torch:1.13
Launcher context deepspeed --num_gpus=2 train.py
May about cuda version,please check nvcc -v. Or up cuda to 11.8.
Hi @Fazziekey - are you still having this issue? And if so, can you try and repro with the latest DeepSpeed?
Closing this issue now for no reply, if anyone is still hitting this, feel free to re-open.