TransformerEngine
TransformerEngine copied to clipboard
Published
20 hours ago •
NVIDIA
NVIDIA
PIP Installation Failed
Hello
I want to install TE using pip:
pip install git+https://github.com/NVIDIA/TransformerEngine.git@stable
But I got the following error during installation:
Collecting git+https://github.com/NVIDIA/TransformerEngine.git@stable
Cloning https://github.com/NVIDIA/TransformerEngine.git (to revision stable) to /tmp/pip-req-build-c6l34itl
Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA/TransformerEngine.git /tmp/pip-req-build-c6l34itl
Running command git checkout -b stable --track origin/stable
Switched to a new branch 'stable'
Branch 'stable' set up to track remote branch 'stable' from 'origin'.
Resolved https://github.com/NVIDIA/TransformerEngine.git to commit bbafb02097e6ca1605c3c0cad84d59dbbcb6e94b
Running command git submodule update --init --recursive -q
Preparing metadata (setup.py) ... done
Collecting pydantic (from transformer-engine==1.2.1+bbafb02)
Downloading pydantic-2.6.0-py3-none-any.whl.metadata (81 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 81.8/81.8 kB 1.1 MB/s eta 0:00:00
Requirement already satisfied: torch in /home/mpngf/.conda/envs/joint_training_221/lib/python3.11/site-packages (from transformer-engine==1.2.1+bbafb02) (2.1.2)
Collecting flash-attn!=2.0.9,!=2.1.0,<=2.3.3,>=1.0.6 (from transformer-engine==1.2.1+bbafb02)
Downloading flash_attn-2.3.3.tar.gz (2.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 11.7 MB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Collecting einops (from flash-attn!=2.0.9,!=2.1.0,<=2.3.3,>=1.0.6->transformer-engine==1.2.1+bbafb02)
Using cached einops-0.7.0-py3-none-any.whl.metadata (13 kB)
Requirement already satisfied: packaging in /home/mpngf/.conda/envs/joint_training_221/lib/python3.11/site-packages (from flash-attn!=2.0.9,!=2.1.0,<=2.3.3,>=1.0.6->transformer-engine==1.2.1+bbafb02) (23.1)
Collecting ninja (from flash-attn!=2.0.9,!=2.1.0,<=2.3.3,>=1.0.6->transformer-engine==1.2.1+bbafb02)
Using cached ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl.metadata (5.3 kB)
Collecting annotated-types>=0.4.0 (from pydantic->transformer-engine==1.2.1+bbafb02)
Using cached annotated_types-0.6.0-py3-none-any.whl.metadata (12 kB)
Collecting pydantic-core==2.16.1 (from pydantic->transformer-engine==1.2.1+bbafb02)
Downloading pydantic_core-2.16.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.5 kB)
Requirement already satisfied: typing-extensions>=4.6.1 in /home/mpngf/.conda/envs/joint_training_221/lib/python3.11/site-packages (from pydantic->transformer-engine==1.2.1+bbafb02) (4.9.0)
Requirement already satisfied: filelock in /home/mpngf/.conda/envs/joint_training_221/lib/python3.11/site-packages (from torch->transformer-engine==1.2.1+bbafb02) (3.13.1)
Requirement already satisfied: sympy in /home/mpngf/.conda/envs/joint_training_221/lib/python3.11/site-packages (from torch->transformer-engine==1.2.1+bbafb02) (1.12)
Requirement already satisfied: networkx in /home/mpngf/.conda/envs/joint_training_221/lib/python3.11/site-packages (from torch->transformer-engine==1.2.1+bbafb02) (3.1)
Requirement already satisfied: jinja2 in /home/mpngf/.conda/envs/joint_training_221/lib/python3.11/site-packages (from torch->transformer-engine==1.2.1+bbafb02) (3.1.2)
Requirement already satisfied: fsspec in /home/mpngf/.conda/envs/joint_training_221/lib/python3.11/site-packages (from torch->transformer-engine==1.2.1+bbafb02) (2023.12.2)
Requirement already satisfied: MarkupSafe>=2.0 in /home/mpngf/.conda/envs/joint_training_221/lib/python3.11/site-packages (from jinja2->torch->transformer-engine==1.2.1+bbafb02) (2.1.3)
Requirement already satisfied: mpmath>=0.19 in /home/mpngf/.conda/envs/joint_training_221/lib/python3.11/site-packages (from sympy->torch->transformer-engine==1.2.1+bbafb02) (1.3.0)
Downloading pydantic-2.6.0-py3-none-any.whl (394 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 394.2/394.2 kB 20.6 MB/s eta 0:00:00
Downloading pydantic_core-2.16.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 34.0 MB/s eta 0:00:00
Using cached annotated_types-0.6.0-py3-none-any.whl (12 kB)
Using cached einops-0.7.0-py3-none-any.whl (44 kB)
Using cached ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl (307 kB)
Building wheels for collected packages: transformer-engine, flash-attn
Building wheel for transformer-engine (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [177 lines of output]
/home/mpngf/.conda/envs/joint_training_221/lib/python3.11/site-packages/setuptools/__init__.py:80: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
!!
********************************************************************************
Requirements should be satisfied by a PEP 517 installer.
If you are using pip, you can try `pip install --use-pep517`.
********************************************************************************
!!
dist.fetch_build_eggs(dist.setup_requires)
running bdist_wheel
/home/mpngf/.conda/envs/joint_training_221/lib/python3.11/site-packages/torch/utils/cpp_extension.py:502: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
warnings.warn(msg.format('we could not find ninja.'))
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-311
creating build/lib.linux-x86_64-cpython-311/transformer_engine
copying transformer_engine/__init__.py -> build/lib.linux-x86_64-cpython-311/transformer_engine
creating build/lib.linux-x86_64-cpython-311/transformer_engine/jax
copying transformer_engine/jax/fused_attn.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax
copying transformer_engine/jax/sharding.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax
copying transformer_engine/jax/dot.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax
copying transformer_engine/jax/fp8.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax
copying transformer_engine/jax/cpp_extensions.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax
copying transformer_engine/jax/mlp.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax
copying transformer_engine/jax/softmax.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax
copying transformer_engine/jax/layernorm.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax
copying transformer_engine/jax/__init__.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax
creating build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
copying transformer_engine/pytorch/numerics_debug.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
copying transformer_engine/pytorch/export.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
copying transformer_engine/pytorch/float8_tensor.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
copying transformer_engine/pytorch/transformer.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
copying transformer_engine/pytorch/utils.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
copying transformer_engine/pytorch/attention.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
copying transformer_engine/pytorch/fp8.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
copying transformer_engine/pytorch/distributed.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
copying transformer_engine/pytorch/te_onnx_extensions.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
copying transformer_engine/pytorch/jit.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
copying transformer_engine/pytorch/softmax.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
copying transformer_engine/pytorch/constants.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
copying transformer_engine/pytorch/__init__.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch
creating build/lib.linux-x86_64-cpython-311/transformer_engine/common
copying transformer_engine/common/utils.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/common
copying transformer_engine/common/recipe.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/common
copying transformer_engine/common/__init__.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/common
creating build/lib.linux-x86_64-cpython-311/transformer_engine/paddle
copying transformer_engine/paddle/utils.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle
copying transformer_engine/paddle/recompute.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle
copying transformer_engine/paddle/fp8_buffer.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle
copying transformer_engine/paddle/fp8.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle
copying transformer_engine/paddle/cpp_extensions.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle
copying transformer_engine/paddle/distributed.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle
copying transformer_engine/paddle/profile.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle
copying transformer_engine/paddle/constants.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle
copying transformer_engine/paddle/__init__.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle
creating build/lib.linux-x86_64-cpython-311/transformer_engine/jax/flax
copying transformer_engine/jax/flax/transformer.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax/flax
copying transformer_engine/jax/flax/module.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax/flax
copying transformer_engine/jax/flax/__init__.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax/flax
creating build/lib.linux-x86_64-cpython-311/transformer_engine/jax/praxis
copying transformer_engine/jax/praxis/transformer.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax/praxis
copying transformer_engine/jax/praxis/module.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax/praxis
copying transformer_engine/jax/praxis/__init__.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/jax/praxis
creating build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/cpp_extensions
copying transformer_engine/pytorch/cpp_extensions/fused_attn.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/cpp_extensions
copying transformer_engine/pytorch/cpp_extensions/normalization.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/cpp_extensions
copying transformer_engine/pytorch/cpp_extensions/transpose.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/cpp_extensions
copying transformer_engine/pytorch/cpp_extensions/cast.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/cpp_extensions
copying transformer_engine/pytorch/cpp_extensions/gemm.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/cpp_extensions
copying transformer_engine/pytorch/cpp_extensions/activation.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/cpp_extensions
copying transformer_engine/pytorch/cpp_extensions/__init__.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/cpp_extensions
creating build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/module
copying transformer_engine/pytorch/module/base.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/module
copying transformer_engine/pytorch/module/_common.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/module
copying transformer_engine/pytorch/module/linear.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/module
copying transformer_engine/pytorch/module/layernorm_mlp.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/module
copying transformer_engine/pytorch/module/layernorm.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/module
copying transformer_engine/pytorch/module/__init__.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/module
copying transformer_engine/pytorch/module/layernorm_linear.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/module
copying transformer_engine/pytorch/module/rmsnorm.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/pytorch/module
creating build/lib.linux-x86_64-cpython-311/transformer_engine/paddle/layer
copying transformer_engine/paddle/layer/base.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle/layer
copying transformer_engine/paddle/layer/transformer.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle/layer
copying transformer_engine/paddle/layer/attention.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle/layer
copying transformer_engine/paddle/layer/softmax.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle/layer
copying transformer_engine/paddle/layer/linear.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle/layer
copying transformer_engine/paddle/layer/layernorm_mlp.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle/layer
copying transformer_engine/paddle/layer/layernorm.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle/layer
copying transformer_engine/paddle/layer/__init__.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle/layer
copying transformer_engine/paddle/layer/layernorm_linear.py -> build/lib.linux-x86_64-cpython-311/transformer_engine/paddle/layer
running build_ext
Building CMake extension transformer_engine
Running command /usr/bin/cmake -S /tmp/pip-req-build-c6l34itl/transformer_engine -B /tmp/tmp7k0z17__ -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/tmp/pip-req-build-c6l34itl/build/lib.linux-x86_64-cpython-311
CMake Error at /usr/share/cmake-3.22/Modules/CMakeDetermineCompilerId.cmake:726 (message):
Compiling the CUDA compiler identification source file
"CMakeCUDACompilerId.cu" failed.
Compiler: CMAKE_CUDA_COMPILER-NOTFOUND
Build flags:
Id flags: -v
The output was:
No such file or directory
Call Stack (most recent call first):
/usr/share/cmake-3.22/Modules/CMakeDetermineCompilerId.cmake:6 (CMAKE_DETERMINE_COMPILER_ID_BUILD)
/usr/share/cmake-3.22/Modules/CMakeDetermineCompilerId.cmake:48 (__determine_compiler_id_test)
/usr/share/cmake-3.22/Modules/CMakeDetermineCUDACompiler.cmake:298 (CMAKE_DETERMINE_COMPILER_ID)
CMakeLists.txt:15 (project)
CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
-- Configuring incomplete, errors occurred!
See also "/tmp/tmp7k0z17__/CMakeFiles/CMakeOutput.log".
See also "/tmp/tmp7k0z17__/CMakeFiles/CMakeError.log".
Traceback (most recent call last):
File "/tmp/pip-req-build-c6l34itl/setup.py", line 353, in _build_cmake
subprocess.run(command, cwd=build_dir, check=True)
File "/home/mpngf/.conda/envs/joint_training_221/lib/python3.11/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/cmake', '-S', '/tmp/pip-req-build-c6l34itl/transformer_engine', '-B', '/tmp/tmp7k0z17__', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_INSTALL_PREFIX=/tmp/pip-req-build-c6l34itl/build/lib.linux-x86_64-cpython-311']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "/tmp/pip-req-build-c6l34itl/setup.py", line 626, in <module>
main()
File "/tmp/pip-req-build-c6l34itl/setup.py", line 611, in main
setuptools.setup(
File "/home/mpngf/.conda/envs/joint_training_221/lib/python3.11/site-packages/setuptools/__init__.py", line 103, in setup
return distutils.core.setup(**attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/mpngf/.conda/envs/joint_training_221/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
return run_commands(dist)
^^^^^^^^^^^^^^^^^^
File "/home/mpngf/.conda/envs/joint_training_221/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
dist.run_commands()
File "/home/mpngf/.conda/envs/joint_training_221/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
self.run_command(cmd)
File "/home/mpngf/.conda/envs/joint_training_221/lib/python3.11/site-packages/setuptools/dist.py", line 989, in run_command
super().run_command(command)
File "/home/mpngf/.conda/envs/joint_training_221/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/home/mpngf/.conda/envs/joint_training_221/lib/python3.11/site-packages/wheel/bdist_wheel.py", line 364, in run
self.run_command("build")
File "/home/mpngf/.conda/envs/joint_training_221/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "/home/mpngf/.conda/envs/joint_training_221/lib/python3.11/site-packages/setuptools/dist.py", line 989, in run_command
super().run_command(command)
File "/home/mpngf/.conda/envs/joint_training_221/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/home/mpngf/.conda/envs/joint_training_221/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 131, in run
self.run_command(cmd_name)
File "/home/mpngf/.conda/envs/joint_training_221/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "/home/mpngf/.conda/envs/joint_training_221/lib/python3.11/site-packages/setuptools/dist.py", line 989, in run_command
super().run_command(command)
File "/home/mpngf/.conda/envs/joint_training_221/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/tmp/pip-req-build-c6l34itl/setup.py", line 383, in run
ext._build_cmake(
File "/tmp/pip-req-build-c6l34itl/setup.py", line 355, in _build_cmake
raise RuntimeError(f"Error when running CMake: {e}")
RuntimeError: Error when running CMake: Command '['/usr/bin/cmake', '-S', '/tmp/pip-req-build-c6l34itl/transformer_engine', '-B', '/tmp/tmp7k0z17__', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_INSTALL_PREFIX=/tmp/pip-req-build-c6l34itl/build/lib.linux-x86_64-cpython-311']' returned non-zero exit status 1.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for transformer-engine
Running setup.py clean for transformer-engine
Building wheel for flash-attn (setup.py) ... done
Created wheel for flash-attn: filename=flash_attn-2.3.3-cp311-cp311-linux_x86_64.whl size=57059098 sha256=499cf51e6036240086c35fd7e49bec42da788d451784ba154b4c654cef4b3510
Stored in directory: /home/mpngf/.cache/pip/wheels/b4/30/9a/5a0c57df68c4836bab04e05a1078d6c378b68c61d38be4f45b
Successfully built flash-attn
Failed to build transformer-engine
ERROR: Could not build wheels for transformer-engine, which is required to install pyproject.toml-based projects
I have A6000 ada GPUs and also, am using conda enviromnet in which I installed:
pytorch 2.1.2 + cuda 12.1
I think the issue might be related to cuda but I dont know how to fix it.
Im running into a similar problem.
pip install --use-pep51 git+https://github.com/NVIDIA/TransformerEngine.git@stable
Collecting git+https://github.com/NVIDIA/TransformerEngine.git@stable
Cloning https://github.com/NVIDIA/TransformerEngine.git (to revision stable) to /tmp/pip-req-build-6bdrrezv
Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA/TransformerEngine.git /tmp/pip-req-build-6bdrrezv
Running command git checkout -b stable --track origin/stable
Switched to a new branch 'stable'
Branch 'stable' set up to track remote branch 'stable' from 'origin'.
Resolved https://github.com/NVIDIA/TransformerEngine.git to commit bbafb02097e6ca1605c3c0cad84d59dbbcb6e94b
Running command git submodule update --init --recursive -q
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing metadata (pyproject.toml) ... done
Collecting pydantic (from transformer_engine==1.2.1+bbafb02)
Using cached pydantic-2.6.1-py3-none-any.whl.metadata (83 kB)
Collecting annotated-types>=0.4.0 (from pydantic->transformer_engine==1.2.1+bbafb02)
Using cached annotated_types-0.6.0-py3-none-any.whl.metadata (12 kB)
Collecting pydantic-core==2.16.2 (from pydantic->transformer_engine==1.2.1+bbafb02)
Using cached pydantic_core-2.16.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.5 kB)
Requirement already satisfied: typing-extensions>=4.6.1 in /usr/local/lib/python3.10/dist-packages (from pydantic->transformer_engine==1.2.1+bbafb02) (4.9.0)
Using cached pydantic-2.6.1-py3-none-any.whl (394 kB)
Using cached pydantic_core-2.16.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB)
Using cached annotated_types-0.6.0-py3-none-any.whl (12 kB)
Building wheels for collected packages: transformer_engine
Building wheel for transformer_engine (pyproject.toml) ... error
error: subprocess-exited-with-error
× Building wheel for transformer_engine (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [171 lines of output]
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-310
creating build/lib.linux-x86_64-cpython-310/transformer_engine
copying transformer_engine/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine
creating build/lib.linux-x86_64-cpython-310/transformer_engine/common
copying transformer_engine/common/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/common
copying transformer_engine/common/recipe.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/common
copying transformer_engine/common/utils.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/common
creating build/lib.linux-x86_64-cpython-310/transformer_engine/jax
copying transformer_engine/jax/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax
copying transformer_engine/jax/cpp_extensions.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax
copying transformer_engine/jax/dot.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax
copying transformer_engine/jax/fp8.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax
copying transformer_engine/jax/fused_attn.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax
copying transformer_engine/jax/layernorm.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax
copying transformer_engine/jax/mlp.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax
copying transformer_engine/jax/sharding.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax
copying transformer_engine/jax/softmax.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax
creating build/lib.linux-x86_64-cpython-310/transformer_engine/paddle
copying transformer_engine/paddle/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/paddle
copying transformer_engine/paddle/constants.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/paddle
copying transformer_engine/paddle/cpp_extensions.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/paddle
copying transformer_engine/paddle/distributed.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/paddle
copying transformer_engine/paddle/fp8.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/paddle
copying transformer_engine/paddle/fp8_buffer.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/paddle
copying transformer_engine/paddle/profile.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/paddle
copying transformer_engine/paddle/recompute.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/paddle
copying transformer_engine/paddle/utils.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/paddle
creating build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
copying transformer_engine/pytorch/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
copying transformer_engine/pytorch/attention.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
copying transformer_engine/pytorch/constants.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
copying transformer_engine/pytorch/distributed.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
copying transformer_engine/pytorch/export.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
copying transformer_engine/pytorch/float8_tensor.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
copying transformer_engine/pytorch/fp8.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
copying transformer_engine/pytorch/jit.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
copying transformer_engine/pytorch/numerics_debug.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
copying transformer_engine/pytorch/softmax.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
copying transformer_engine/pytorch/te_onnx_extensions.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
copying transformer_engine/pytorch/transformer.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
copying transformer_engine/pytorch/utils.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
creating build/lib.linux-x86_64-cpython-310/transformer_engine/jax/flax
copying transformer_engine/jax/flax/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax/flax
copying transformer_engine/jax/flax/module.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax/flax
copying transformer_engine/jax/flax/transformer.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax/flax
creating build/lib.linux-x86_64-cpython-310/transformer_engine/jax/praxis
copying transformer_engine/jax/praxis/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax/praxis
copying transformer_engine/jax/praxis/module.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax/praxis
copying transformer_engine/jax/praxis/transformer.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax/praxis
creating build/lib.linux-x86_64-cpython-310/transformer_engine/paddle/layer
copying transformer_engine/paddle/layer/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/paddle/layer
copying transformer_engine/paddle/layer/attention.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/paddle/layer
copying transformer_engine/paddle/layer/base.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/paddle/layer
copying transformer_engine/paddle/layer/layernorm.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/paddle/layer
copying transformer_engine/paddle/layer/layernorm_linear.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/paddle/layer
copying transformer_engine/paddle/layer/layernorm_mlp.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/paddle/layer
copying transformer_engine/paddle/layer/linear.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/paddle/layer
copying transformer_engine/paddle/layer/softmax.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/paddle/layer
copying transformer_engine/paddle/layer/transformer.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/paddle/layer
creating build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/cpp_extensions
copying transformer_engine/pytorch/cpp_extensions/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/cpp_extensions
copying transformer_engine/pytorch/cpp_extensions/activation.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/cpp_extensions
copying transformer_engine/pytorch/cpp_extensions/cast.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/cpp_extensions
copying transformer_engine/pytorch/cpp_extensions/fused_attn.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/cpp_extensions
copying transformer_engine/pytorch/cpp_extensions/gemm.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/cpp_extensions
copying transformer_engine/pytorch/cpp_extensions/normalization.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/cpp_extensions
copying transformer_engine/pytorch/cpp_extensions/transpose.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/cpp_extensions
creating build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/module
copying transformer_engine/pytorch/module/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/module
copying transformer_engine/pytorch/module/_common.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/module
copying transformer_engine/pytorch/module/base.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/module
copying transformer_engine/pytorch/module/layernorm.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/module
copying transformer_engine/pytorch/module/layernorm_linear.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/module
copying transformer_engine/pytorch/module/layernorm_mlp.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/module
copying transformer_engine/pytorch/module/linear.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/module
copying transformer_engine/pytorch/module/rmsnorm.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/module
running build_ext
-- The CUDA compiler identification is NVIDIA 12.1.105
-- The CXX compiler identification is GNU 11.4.0
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found CUDAToolkit: /usr/local/cuda/targets/x86_64-linux/include (found version "12.1.105")
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- cudnn not found.
-- cudnn_adv_infer not found.
-- cudnn_adv_train not found.
-- cudnn_cnn_infer not found.
-- cudnn_cnn_train not found.
-- cudnn_ops_infer not found.
-- cudnn_ops_train not found.
CMake Error at /tmp/pip-build-env-a74ikyx0/normal/local/lib/python3.10/dist-packages/cmake/data/share/cmake-3.28/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
Could NOT find CUDNN (missing: CUDNN_LIBRARY)
Call Stack (most recent call first):
/tmp/pip-build-env-a74ikyx0/normal/local/lib/python3.10/dist-packages/cmake/data/share/cmake-3.28/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
cmake/FindCUDNN.cmake:46 (find_package_handle_standard_args)
CMakeLists.txt:24 (find_package)
-- Configuring incomplete, errors occurred!
Building CMake extension transformer_engine
Running command /tmp/pip-build-env-a74ikyx0/normal/local/lib/python3.10/dist-packages/cmake/data/bin/cmake -S /tmp/pip-req-build-6bdrrezv/transformer_engine -B /tmp/tmpscbva33l -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/tmp/pip-req-build-6bdrrezv/build/lib.linux-x86_64-cpython-310 -GNinja
Traceback (most recent call last):
File "<string>", line 353, in _build_cmake
File "/usr/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/tmp/pip-build-env-a74ikyx0/normal/local/lib/python3.10/dist-packages/cmake/data/bin/cmake', '-S', '/tmp/pip-req-build-6bdrrezv/transformer_engine', '-B', '/tmp/tmpscbva33l', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_INSTALL_PREFIX=/tmp/pip-req-build-6bdrrezv/build/lib.linux-x86_64-cpython-310', '-GNinja']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
main()
File "/usr/local/lib/python3.10/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/usr/local/lib/python3.10/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
return _build_backend().build_wheel(wheel_directory, config_settings,
File "/tmp/pip-build-env-a74ikyx0/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 404, in build_wheel
return self._build_with_temp_dir(
File "/tmp/pip-build-env-a74ikyx0/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 389, in _build_with_temp_dir
self.run_setup()
File "/tmp/pip-build-env-a74ikyx0/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 480, in run_setup
super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
File "/tmp/pip-build-env-a74ikyx0/overlay/local/lib/python3.10/dist-packages/setuptools/build_meta.py", line 311, in run_setup
exec(code, locals())
File "<string>", line 626, in <module>
File "<string>", line 611, in main
File "/tmp/pip-build-env-a74ikyx0/overlay/local/lib/python3.10/dist-packages/setuptools/__init__.py", line 103, in setup
return distutils.core.setup(**attrs)
File "/tmp/pip-build-env-a74ikyx0/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/core.py", line 185, in setup
return run_commands(dist)
File "/tmp/pip-build-env-a74ikyx0/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/core.py", line 201, in run_commands
dist.run_commands()
File "/tmp/pip-build-env-a74ikyx0/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 969, in run_commands
self.run_command(cmd)
File "/tmp/pip-build-env-a74ikyx0/overlay/local/lib/python3.10/dist-packages/setuptools/dist.py", line 963, in run_command
super().run_command(command)
File "/tmp/pip-build-env-a74ikyx0/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/tmp/pip-build-env-a74ikyx0/normal/local/lib/python3.10/dist-packages/wheel/bdist_wheel.py", line 368, in run
self.run_command("build")
File "/tmp/pip-build-env-a74ikyx0/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "/tmp/pip-build-env-a74ikyx0/overlay/local/lib/python3.10/dist-packages/setuptools/dist.py", line 963, in run_command
super().run_command(command)
File "/tmp/pip-build-env-a74ikyx0/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/tmp/pip-build-env-a74ikyx0/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/command/build.py", line 131, in run
self.run_command(cmd_name)
File "/tmp/pip-build-env-a74ikyx0/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "/tmp/pip-build-env-a74ikyx0/overlay/local/lib/python3.10/dist-packages/setuptools/dist.py", line 963, in run_command
super().run_command(command)
File "/tmp/pip-build-env-a74ikyx0/overlay/local/lib/python3.10/dist-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "<string>", line 383, in run
File "<string>", line 355, in _build_cmake
RuntimeError: Error when running CMake: Command '['/tmp/pip-build-env-a74ikyx0/normal/local/lib/python3.10/dist-packages/cmake/data/bin/cmake', '-S', '/tmp/pip-req-build-6bdrrezv/transformer_engine', '-B', '/tmp/tmpscbva33l', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_INSTALL_PREFIX=/tmp/pip-req-build-6bdrrezv/build/lib.linux-x86_64-cpython-310', '-GNinja']' returned non-zero exit status 1.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for transformer_engine
Failed to build transformer_engine
ERROR: Could not build wheels for transformer_engine, which is required to install pyproject.toml-based projects
root@7d9c74e72ac3:/usr/local/cuda/lib64#
Downloaded
https://developer.nvidia.com/downloads/compute/cudnn/secure/8.9.7/local_installers/12.x/cudnn-linux-x86_64-8.9.7.29_cuda12-archive.tar.xz/
ran
tar -xvf cudnn-linux-x86_64-8.9.7.29_cuda12-archive.tar.xz && cp cudnn-linux-x86_64-8.9.7.29_cuda12-archive/lib/libcudnn* /usr/local/cuda/lib64/ && cp cudnn-linux-x86_64-8.9.7.29_cuda12-archive/include/cudnn*.h /usr/local/cuda/include/ && ldconfig /usr/local/cuda/lib64 && cat /usr/local/cuda/include/cudnn_version.h | grep CUDNN_MAJOR -A 2 && pip install --use-pep51 git+https://github.com/NVIDIA/TransformerEngine.git@stable
works.
Since my pytorch already uses cudnn there mustve been another installation though.
root@7d9c74e72ac3:/usr/local/cuda/lib64# find / -name "libcudnn.so*" 2>/dev/null
/usr/local/lib/python3.10/dist-packages/nvidia/cudnn/lib/libcudnn.so.8
root@7d9c74e72ac3:/usr/local/cuda/lib64#
Maybe the build tool could check here too.
Now execution fails because "transformerengine_extras" can't be found...
@mahdip72 It looks CMake is having trouble finding your C++ compiler and your CUDA installation. Can you try setting the CXX and CUDA_PATH environment variables?
@markusheimerl The best way to specify custom cuDNN installs is by setting the CUDNN_PATH environment variable. As for the runtime failure, I assume it's not correctly importing transformer_engine_extensions? Can you check if the extensions .so file installed (should look something like transformer_engine_extensions.cpython-310-x86_64-linux-gnu.so)? If it was, then it should just be a matter of setting PYTHONPATH so Python can find it. Otherwise, we'll need to debug how Transformer Engine got past the build process without complaining (maybe it didn't find PyTorch and skipped building the extensions?).
