dolly icon indicating copy to clipboard operation
dolly copied to clipboard
verified publisher icon databrickslabs

RuntimeError: Error building extension 'fused_adam'

Open chintan-donda opened this issue 2 years ago • 3 comments

While training the Dolly model on V100 single GPU, I'm getting the below error:

Detected CUDA files, patching ldflags
Emitting ninja build file /home/users/abc/.cache/torch_extensions/py38_cu117/fused_adam/build.ninja...
Building extension module fused_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/3] c++ -MMD -MF fused_adam_frontend.o.d -DTORCH_EXTENSION_NAME=fused_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/users/abc/venv/lib/python3.8/site-packages/deepspeed/ops/csrc/includes -I/home/users/abc/venv/lib/python3.8/site-packages/deepspeed/ops/csrc/adam -isystem /home/users/abc/venv/lib/python3.8/site-packages/torch/include -isystem /home/users/abc/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/users/abc/venv/lib/python3.8/site-packages/torch/include/TH -isystem /home/users/abc/venv/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -std=c++14 -g -Wno-reorder -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -c /home/users/abc/venv/lib/python3.8/site-packages/deepspeed/ops/csrc/adam/fused_adam_frontend.cpp -o fused_adam_frontend.o 
FAILED: fused_adam_frontend.o 
c++ -MMD -MF fused_adam_frontend.o.d -DTORCH_EXTENSION_NAME=fused_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/users/abc/venv/lib/python3.8/site-packages/deepspeed/ops/csrc/includes -I/home/users/abc/venv/lib/python3.8/site-packages/deepspeed/ops/csrc/adam -isystem /home/users/abc/venv/lib/python3.8/site-packages/torch/include -isystem /home/users/abc/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/users/abc/venv/lib/python3.8/site-packages/torch/include/TH -isystem /home/users/abc/venv/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -std=c++14 -g -Wno-reorder -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -c /home/users/abc/venv/lib/python3.8/site-packages/deepspeed/ops/csrc/adam/fused_adam_frontend.cpp -o fused_adam_frontend.o 
In file included from /home/users/abc/venv/lib/python3.8/site-packages/torch/include/torch/csrc/Device.h:4:0,
                 from /home/users/abc/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include/torch/python.h:8,
                 from /home/users/abc/venv/lib/python3.8/site-packages/torch/include/torch/extension.h:6,
                 from /home/users/abc/venv/lib/python3.8/site-packages/deepspeed/ops/csrc/adam/fused_adam_frontend.cpp:1:
/home/users/abc/venv/lib/python3.8/site-packages/torch/include/torch/csrc/python_headers.h:12:10: fatal error: Python.h: No such file or directory
 #include <Python.h>
          ^~~~~~~~~~
compilation terminated.

[2/3] /usr/local/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=fused_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/home/users/abc/venv/lib/python3.8/site-packages/deepspeed/ops/csrc/includes -I/home/users/abc/venv/lib/python3.8/site-packages/deepspeed/ops/csrc/adam -isystem /home/users/abc/venv/lib/python3.8/site-packages/torch/include -isystem /home/users/abc/venv/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/users/abc/venv/lib/python3.8/site-packages/torch/include/TH -isystem /home/users/abc/venv/lib/python3.8/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_70,code=compute_70 -gencode=arch=compute_70,code=sm_70 --compiler-options '-fPIC' -O3 -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -lineinfo --use_fast_math -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_70,code=compute_70 -std=c++14 -c /home/users/abc/venv/lib/python3.8/site-packages/deepspeed/ops/csrc/adam/multi_tensor_adam.cu -o multi_tensor_adam.cuda.o 
ninja: build stopped: subcommand failed.
2023-05-11 20:12:51 ERROR [__main__] main failed
Traceback (most recent call last):
  File "/home/users/abc/venv/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1900, in _run_ninja_build
    subprocess.run(
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/users/abc/dolly/training/trainer.py", line 329, in <module>
    main()
  File "/home/users/abc/venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/users/abc/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/users/abc/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/users/abc/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/users/abc/dolly/training/trainer.py", line 321, in main
    train(**kwargs)
  File "/home/users/abc/dolly/training/trainer.py", line 276, in train
    trainer.train()
  File "/home/users/abc/venv/lib/python3.8/site-packages/transformers/trainer.py", line 1664, in train
    return inner_training_loop(
  File "/home/users/abc/venv/lib/python3.8/site-packages/transformers/trainer.py", line 1741, in _inner_training_loop
    deepspeed_engine, optimizer, lr_scheduler = deepspeed_init(
  File "/home/users/abc/venv/lib/python3.8/site-packages/transformers/deepspeed.py", line 378, in deepspeed_init
    deepspeed_engine, optimizer, _, lr_scheduler = deepspeed.initialize(**kwargs)
  File "/home/users/abc/venv/lib/python3.8/site-packages/deepspeed/__init__.py", line 125, in initialize
    engine = DeepSpeedEngine(args=args,
  File "/home/users/abc/venv/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 340, in __init__
    self._configure_optimizer(optimizer, model_parameters)
  File "/home/users/abc/venv/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 1283, in _configure_optimizer
    basic_optimizer = self._configure_basic_optimizer(model_parameters)
  File "/home/users/abc/venv/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 1360, in _configure_basic_optimizer
    optimizer = FusedAdam(
  File "/home/users/abc/venv/lib/python3.8/site-packages/deepspeed/ops/adam/fused_adam.py", line 73, in __init__
    fused_adam_cuda = FusedAdamBuilder().load()
  File "/home/users/abc/venv/lib/python3.8/site-packages/deepspeed/ops/op_builder/builder.py", line 485, in load
    return self.jit_load(verbose)
  File "/home/users/abc/venv/lib/python3.8/site-packages/deepspeed/ops/op_builder/builder.py", line 520, in jit_load
    op_module = load(
  File "/home/users/abc/venv/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1284, in load
    return _jit_compile(
  File "/home/users/abc/venv/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1508, in _jit_compile
    _write_ninja_file_and_build_library(
  File "/home/users/abc/venv/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1623, in _write_ninja_file_and_build_library
    _run_ninja_build(
  File "/home/users/abc/venv/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1916, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error building extension 'fused_adam'
Traceback (most recent call last):
  File "/home/users/abc/venv/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1900, in _run_ninja_build
    subprocess.run(
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/users/abc/dolly/training/trainer.py", line 329, in <module>
    main()
  File "/home/users/abc/venv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/users/abc/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/users/abc/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/users/abc/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/users/abc/dolly/training/trainer.py", line 321, in main
    train(**kwargs)
  File "/home/users/abc/dolly/training/trainer.py", line 276, in train
    trainer.train()
  File "/home/users/abc/venv/lib/python3.8/site-packages/transformers/trainer.py", line 1664, in train
    return inner_training_loop(
  File "/home/users/abc/venv/lib/python3.8/site-packages/transformers/trainer.py", line 1741, in _inner_training_loop
    deepspeed_engine, optimizer, lr_scheduler = deepspeed_init(
  File "/home/users/abc/venv/lib/python3.8/site-packages/transformers/deepspeed.py", line 378, in deepspeed_init
    deepspeed_engine, optimizer, _, lr_scheduler = deepspeed.initialize(**kwargs)
  File "/home/users/abc/venv/lib/python3.8/site-packages/deepspeed/__init__.py", line 125, in initialize
    engine = DeepSpeedEngine(args=args,
  File "/home/users/abc/venv/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 340, in __init__
    self._configure_optimizer(optimizer, model_parameters)
  File "/home/users/abc/venv/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 1283, in _configure_optimizer
    basic_optimizer = self._configure_basic_optimizer(model_parameters)
  File "/home/users/abc/venv/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 1360, in _configure_basic_optimizer
    optimizer = FusedAdam(
  File "/home/users/abc/venv/lib/python3.8/site-packages/deepspeed/ops/adam/fused_adam.py", line 73, in __init__
    fused_adam_cuda = FusedAdamBuilder().load()
  File "/home/users/abc/venv/lib/python3.8/site-packages/deepspeed/ops/op_builder/builder.py", line 485, in load
    return self.jit_load(verbose)
  File "/home/users/abc/venv/lib/python3.8/site-packages/deepspeed/ops/op_builder/builder.py", line 520, in jit_load
    op_module = load(
  File "/home/users/abc/venv/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1284, in load
    return _jit_compile(
  File "/home/users/abc/venv/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1508, in _jit_compile
    _write_ninja_file_and_build_library(
  File "/home/users/abc/venv/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1623, in _write_ninja_file_and_build_library
    _run_ninja_build(
  File "/home/users/abc/venv/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1916, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error building extension 'fused_adam'
[2023-05-11 20:12:52,931] [INFO] [launch.py:318:sigkill_handler] Killing subprocess 62415

Command to run:

deepspeed --num_gpus=1 --module training.trainer --deepspeed $deepspeed_config --epochs 1 --local-output-dir $local_output_dir --dbfs-output-dir "" --per-device-train-batch-size 4  --per-device-eval-batch-size 8 --lr 1e-5 --warmup-steps 1

Environment details:

Python 3.8.16
NVIDIA-SMI 515.86.01
Driver Version: 515.86.01
CUDA Version: 11.7

deepspeed==0.8.3
ninja==1.11.1
nvidia-cublas-cu11==11.10.3.66
nvidia-cuda-nvrtc-cu11==11.7.99
nvidia-cuda-runtime-cu11==11.7.99
nvidia-cudnn-cu11==8.5.0.96
torch==1.13.1
torchvision==0.14.1
transformers==4.29.0

Any help?

chintan-donda avatar May 11 '23 14:05 chintan-donda

Hm, never seen that one. I am not sure if the nvidia-cublas-cu11 python packages you installed are what you want for the nvidia libs. have you installed the OS libs as shown at https://github.com/databrickslabs/dolly/blob/master/train_dolly.py#L53 ?

srowen avatar May 12 '23 12:05 srowen

@srowen Did that. But same issue.

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/libcusparse-dev-11-7_11.7.3.50-1_amd64.deb -O /tmp/libcusparse-dev-11-7_11.7.3.50-1_amd64.deb
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/libcublas-dev-11-7_11.10.1.25-1_amd64.deb -O /tmp/libcublas-dev-11-7_11.10.1.25-1_amd64.deb
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/libcusolver-dev-11-7_11.4.0.1-1_amd64.deb -O /tmp/libcusolver-dev-11-7_11.4.0.1-1_amd64.deb
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/libcurand-dev-11-7_10.2.10.91-1_amd64.deb -O /tmp/libcurand-dev-11-7_10.2.10.91-1_amd64.deb
dpkg -i /tmp/libcusparse-dev-11-7_11.7.3.50-1_amd64.deb
dpkg -i /tmp/libcublas-dev-11-7_11.10.1.25-1_amd64.deb
dpkg -i /tmp/libcusolver-dev-11-7_11.4.0.1-1_amd64.deb
dpkg -i /tmp/libcurand-dev-11-7_10.2.10.91-1_amd64.deb

chintan-donda avatar May 13 '23 06:05 chintan-donda

Do you have the python3-dev OS package installed?

srowen avatar May 13 '23 11:05 srowen

@srowen Yes, python3-dev OS package is installed. And still the same issue. FYI, issue coming in both EleutherAI/pythia-2.8b and EleutherAI/pythia-6.9b. Didn't try the other variants due to machine limitations.

I see others have also faced the similar issue.

chintan-donda avatar May 15 '23 06:05 chintan-donda

I've seen a failure to build fused_adam because of missing libraries, but yours is different than what I've seen and that issue. You're somehow missing the python dev headers. Maybe paths are wrong? it'll be specific to your env.

srowen avatar May 15 '23 12:05 srowen

I'm still facing this issue.

Here is the ds_report:

--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.1
 [WARNING]  using untested triton version (2.1.0+7d1a95b046), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
utils .................. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/home/users/chintan/venv/lib/python3.8/site-packages/torch']
torch version .................... 2.1.0.dev20230516+cu117
deepspeed install path ........... ['/home/users/chintan/venv/lib/python3.8/site-packages/deepspeed']
deepspeed info ................... 0.9.2, unknown, unknown
torch cuda version ............... 11.7
torch hip version ................ None
nvcc version ..................... 10.0
deepspeed wheel compiled w. ...... torch 1.13, cuda 11.7

pip freeze:

accelerate==0.19.0
aiohttp==3.8.4
aiosignal==1.3.1
async-timeout==4.0.2
attrs==22.2.0
awscli==1.27.133
bitsandbytes==0.38.1
boto3==1.26.133
botocore==1.29.133
certifi==2022.12.7
charset-normalizer==2.1.1
click==8.1.3
cloudpickle==2.2.1
cmake==3.25.0
colorama==0.4.4
contextlib2==21.6.0
coverage==7.2.5
dataclasses-json==0.5.7
datasets==2.12.0
deepspeed==0.9.2
dill==0.3.6
docutils==0.16
filelock==3.9.0
frozenlist==1.3.3
fsspec==2023.4.0
google-pasta==0.2.0
greenlet==2.0.2
hjson==3.1.0
huggingface-hub==0.14.1
idna==3.4
importlib-metadata==4.13.0   
importlib-resources==5.12.0  
iniconfig==2.0.0
Jinja2==3.1.2
jmespath==1.0.1
jsonschema==4.17.3
langchain==0.0.165
lit==15.0.7
MarkupSafe==2.1.2
marshmallow==3.19.0
marshmallow-enum==1.5.1
mpmath==1.2.1
multidict==6.0.4
multiprocess==0.70.14
mypy-extensions==1.0.0
networkx==3.0rc1
ninja==1.11.1
numexpr==2.8.4
numpy==1.24.1
nvidia-cublas-cu11==11.10.3.66
nvidia-cuda-nvrtc-cu11==11.7.99
nvidia-cuda-runtime-cu11==11.7.99
nvidia-cudnn-cu11==8.5.0.96  
openapi-schema-pydantic==1.2.4
packaging==23.1
pandas==2.0.1
pathos==0.3.0
Pillow==9.3.0
pkgutil_resolve_name==1.3.10 
platformdirs==3.5.1
pluggy==1.0.0
pox==0.3.2
ppft==1.7.6.6
protobuf==3.20.3
protobuf3-to-dict==0.1.5
psutil==5.9.5
py==1.11.0
py-cpuinfo==9.0.0
pyarrow==12.0.0
pyasn1==0.5.0
pydantic==1.10.7
pyrsistent==0.19.3
pytest==7.1.2
pytest-cov==3.0.0
python-dateutil==2.8.2
pytorch-triton==2.1.0+7d1a95b046
pytz==2023.3
PyYAML==5.4.1
regex==2023.5.5
requests==2.28.1
responses==0.18.0
rsa==4.7.2
s3transfer==0.6.1
sagemaker==2.154.0
schema==0.7.5
scipy==1.10.1
sentencepiece==0.1.99
six==1.16.0
smdebug-rulesconfig==1.0.1
SQLAlchemy==2.0.13
sympy==1.11.1
tblib==1.7.0
tenacity==8.2.2
tensorboardX==2.6
tokenizers==0.13.3
tomli==2.0.1
torch==2.1.0.dev20230516+cu117
torchaudio==2.1.0.dev20230516+cu117
torchvision==0.16.0.dev20230516+cu117
tqdm==4.65.0
transformers @ git+https://github.com/huggingface/transformers.git@d765717c76026281f2fb27ddc44fa3636306bb48
triton==2.0.0
typing-inspect==0.8.0
typing_extensions==4.4.0
tzdata==2023.3
urllib3==1.26.13
xxhash==3.2.0
yarl==1.9.2
zipp==3.15.0

NVCC version:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130

nvidia-smi details:

Wed May 17 15:37:55 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.86.01    Driver Version: 515.86.01    CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+

chintan-donda avatar May 17 '23 10:05 chintan-donda

Wrong torch version - use the versions in requirements.txt. It hints at this: [WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.1

srowen avatar May 17 '23 11:05 srowen