Install from source stuck on 3.0.0

[gautierronan]

Hi, I wanted to try out the tl.split and tl.join functions which only seem available on triton 3.0.0. However, installing from source as per the README doesn't seem to work. The installation has been stuck on running build_ext for tens of minutes.

Any idea?

>> pip install --verbose -e python
Using pip 24.0 from /home/rgautier/.pyenv/versions/3.11.8/lib/python3.11/site-packages/pip (python 3.11)
Obtaining file:///home/rgautier/triton/python
  Running command pip subprocess to install build dependencies
  Collecting setuptools>=40.8.0
    Using cached setuptools-69.4.0-py3-none-any.whl.metadata (6.2 kB)
  Collecting wheel
    Using cached wheel-0.43.0-py3-none-any.whl.metadata (2.2 kB)
  Collecting cmake>=3.18
    Using cached cmake-3.29.2-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.1 kB)
  Collecting ninja>=1.11.1
    Using cached ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl.metadata (5.3 kB)
  Using cached setuptools-69.4.0-py3-none-any.whl (823 kB)
  Using cached wheel-0.43.0-py3-none-any.whl (65 kB)
  Using cached cmake-3.29.2-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (26.7 MB)
  Using cached ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl (307 kB)
  Installing collected packages: ninja, wheel, setuptools, cmake
  Successfully installed cmake-3.29.2 ninja-1.11.1.1 setuptools-69.4.0 wheel-0.43.0
  Installing build dependencies ... done
  Running command Checking if build backend supports build_editable
  Checking if build backend supports build_editable ... done
  Running command Getting requirements to build editable
  copy /home/rgautier/.triton/nvidia/bin/ptxas to /home/rgautier/triton/python/../third_party/nvidia/backend/bin/ptxas ...
  copy /home/rgautier/.triton/nvidia/bin/cuobjdump to /home/rgautier/triton/python/../third_party/nvidia/backend/bin/cuobjdump ...
  copy /home/rgautier/.triton/nvidia/bin/nvdisasm to /home/rgautier/triton/python/../third_party/nvidia/backend/bin/nvdisasm ...
  running egg_info
  writing triton.egg-info/PKG-INFO
  writing dependency_links to triton.egg-info/dependency_links.txt
  writing requirements to triton.egg-info/requires.txt
  writing top-level names to triton.egg-info/top_level.txt
  reading manifest file 'triton.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  writing manifest file 'triton.egg-info/SOURCES.txt'
  Getting requirements to build editable ... done
  Running command Preparing editable metadata (pyproject.toml)
  copy /home/rgautier/.triton/nvidia/bin/ptxas to /home/rgautier/triton/python/../third_party/nvidia/backend/bin/ptxas ...
  copy /home/rgautier/.triton/nvidia/bin/cuobjdump to /home/rgautier/triton/python/../third_party/nvidia/backend/bin/cuobjdump ...
  copy /home/rgautier/.triton/nvidia/bin/nvdisasm to /home/rgautier/triton/python/../third_party/nvidia/backend/bin/nvdisasm ...
  running dist_info
  creating /tmp/pip-modern-metadata-3k02gtoz/triton.egg-info
  writing /tmp/pip-modern-metadata-3k02gtoz/triton.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-modern-metadata-3k02gtoz/triton.egg-info/dependency_links.txt
  writing requirements to /tmp/pip-modern-metadata-3k02gtoz/triton.egg-info/requires.txt
  writing top-level names to /tmp/pip-modern-metadata-3k02gtoz/triton.egg-info/top_level.txt
  writing manifest file '/tmp/pip-modern-metadata-3k02gtoz/triton.egg-info/SOURCES.txt'
  reading manifest file '/tmp/pip-modern-metadata-3k02gtoz/triton.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  writing manifest file '/tmp/pip-modern-metadata-3k02gtoz/triton.egg-info/SOURCES.txt'
  creating '/tmp/pip-modern-metadata-3k02gtoz/triton-3.0.0.dist-info'
  Preparing editable metadata (pyproject.toml) ... done
Requirement already satisfied: filelock in /home/rgautier/.pyenv/versions/3.11.8/lib/python3.11/site-packages (from triton==3.0.0) (3.13.1)
Building wheels for collected packages: triton
  Running command Building editable for triton (pyproject.toml)
  copy /home/rgautier/.triton/nvidia/bin/ptxas to /home/rgautier/triton/python/../third_party/nvidia/backend/bin/ptxas ...
  copy /home/rgautier/.triton/nvidia/bin/cuobjdump to /home/rgautier/triton/python/../third_party/nvidia/backend/bin/cuobjdump ...
  copy /home/rgautier/.triton/nvidia/bin/nvdisasm to /home/rgautier/triton/python/../third_party/nvidia/backend/bin/nvdisasm ...
  running editable_wheel
  creating /tmp/pip-wheel-v165jqjq/.tmp-es_jn_x7/triton.egg-info
  writing /tmp/pip-wheel-v165jqjq/.tmp-es_jn_x7/triton.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-wheel-v165jqjq/.tmp-es_jn_x7/triton.egg-info/dependency_links.txt
  writing requirements to /tmp/pip-wheel-v165jqjq/.tmp-es_jn_x7/triton.egg-info/requires.txt
  writing top-level names to /tmp/pip-wheel-v165jqjq/.tmp-es_jn_x7/triton.egg-info/top_level.txt
  writing manifest file '/tmp/pip-wheel-v165jqjq/.tmp-es_jn_x7/triton.egg-info/SOURCES.txt'
  reading manifest file '/tmp/pip-wheel-v165jqjq/.tmp-es_jn_x7/triton.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  writing manifest file '/tmp/pip-wheel-v165jqjq/.tmp-es_jn_x7/triton.egg-info/SOURCES.txt'
  creating '/tmp/pip-wheel-v165jqjq/.tmp-es_jn_x7/triton-3.0.0.dist-info'
  creating /tmp/pip-wheel-v165jqjq/.tmp-es_jn_x7/triton-3.0.0.dist-info/WHEEL
  running build_py
  running build_ext

Thanks for the help.

EDIT: Also tried on a fresh installation, and same thing happening.

[fkouteib]

hey @gautierronan I can't speculate on the cause of the build hang issue. The point of failure is right when Cmake kicks off the compiler identification process. It would look like this

running build_py running build_ext CMake Deprecation Warning at CMakeLists.txt:6 (cmake_policy): The OLD behavior for policy CMP0116 will be removed from a future version of CMake. The cmake-policies(7) manual explains that the OLD behaviors of all policies are deprecated and that a policy should be set to OLD only under specific short-term circumstances. Projects should be ported to the NEW behavior and not rely on setting a policy to OLD. -- The C compiler identification is AppleClang 15.0.0.15000309 -- The CXX compiler identification is AppleClang 15.0.0.15000309 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- Looking for histedit.h -- Looking for histedit.h - found

So if you want to pursue building from source, verify you have GCC/G++ or Clang/Clang++/LLD installed on your machine as a starting point.

If you just need a Triton install, there are now nightly builds of triton 3.0 available here. Last one built on 4/14/2024.

You can install it with pip install -U --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/Triton-Nightly/pypi/simple/ triton-nightly

[jsonlee0x02]

I also encountered a similar problem when compiling from source code:

ImportError: triton/python/triton/_C/libtriton.so: undefined symbol: LLVMInitializeSparcTarget

seems something is wrong with LLVM backend?

[jsonlee0x02]

When I try pip install -U --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/Triton-Nightly/pypi/simple/ triton-nightly, and run the tutorial vector-add example,

cd triton/python/tutorials
python 01-vector-add.py

triger the errors:

File "miniconda3/lib/python3.10/site-packages/triton/compiler/compiler.py", line 369, in _init_handles
    self.module, self.function, self.n_regs, self.n_spills = driver.active.utils.load_binary(
RuntimeError: Triton Error [CUDA]: device kernel image is invalid

[fkouteib]

I also encountered a similar problem when compiling from source code:

ImportError: triton/python/triton/_C/libtriton.so: undefined symbol: LLVMInitializeSparcTarget

seems something is wrong with LLVM backend?

@jsonlee0x02 I don't think it is the same issue. The original errors were showing install on x86-64 target. I am inferring from the missing symbol that you are running on a SPARC (HW arch) system running some flavor of Linux. So even if you build your own LLVM target from source, you need additional modifications here to make use of those variants of those libs. Otherwise, you are telling cmake to link against x86 LLVM libs here, instead of the SPARC LLVM lib binaries you compiled. So at runtime it's not gonna find the SPARC matching symbols in the package.

For the second comment, there is not much to go on to help you debug the issue but I am guessing it's a Linux x86-64 system if you were able to install triton nightly build. You should probably separate it into a different issue, and capture your system config HW (cpu arch, which GPU and compute capability), GPU device driver version, CUDA Toolkit version, OS flavor and version, PyTorch version, exact Triton nightly version installed (run 'pip show triton' to get the v3.0.0 suffix) as the min required info as part of repro recipe.

EDIT: I misspoke on one thing in the last paragraph. While the cmake issue I pointed out is valid. It's not the root cause (assuming steps in readme were followed). There is a bug in build script that would cause linux builds to default to x86-64 (if not aarch64 target) llvm build even if user is building for another arch on Linux and provided env variables pointing to their local build. I put up a PR for that #3719.

[jsonlee0x02]

I also encountered a similar problem when compiling from source code:

ImportError: triton/python/triton/_C/libtriton.so: undefined symbol: LLVMInitializeSparcTarget

seems something is wrong with LLVM backend?

@jsonlee0x02 I don't think it is the same issue. The original errors were showing install on x86-64 target. I am inferring from the missing symbol that you are running on a SPARC (HW arch) system running some flavor of Linux. So even if you build your own LLVM target from source, you need additional modifications here to make use of those variants of those libs. Otherwise, you are telling cmake to link against x86 LLVM libs here, instead of the SPARC LLVM lib binaries you compiled. So at runtime it's not gonna find the SPARC matching symbols in the package.

For the second comment, there is not much to go on to help you debug the issue but I am guessing it's a Linux x86-64 system if you were able to install triton nightly build. You should probably separate it into a different issue, and capture your system config HW (cpu arch, which GPU and compute capability), GPU device driver version, CUDA Toolkit version, OS flavor and version, PyTorch version, exact Triton nightly version installed (run 'pip show triton' to get the v3.0.0 suffix) as the min required info as part of repro recipe.

EDIT: I misspoke on one thing in the last paragraph. While the cmake issue I pointed out is valid. It's not the root cause (assuming steps in readme were followed). There is a bug in build script that would cause linux builds to default to x86-64 (if not aarch64 target) llvm build even if user is building for another arch on Linux and provided env variables pointing to their local build. I put up a PR for that #3719.

I also encountered a similar problem when compiling from source code:

ImportError: triton/python/triton/_C/libtriton.so: undefined symbol: LLVMInitializeSparcTarget

seems something is wrong with LLVM backend?

@jsonlee0x02 I don't think it is the same issue. The original errors were showing install on x86-64 target. I am inferring from the missing symbol that you are running on a SPARC (HW arch) system running some flavor of Linux. So even if you build your own LLVM target from source, you need additional modifications here to make use of those variants of those libs. Otherwise, you are telling cmake to link against x86 LLVM libs here, instead of the SPARC LLVM lib binaries you compiled. So at runtime it's not gonna find the SPARC matching symbols in the package.

For the second comment, there is not much to go on to help you debug the issue but I am guessing it's a Linux x86-64 system if you were able to install triton nightly build. You should probably separate it into a different issue, and capture your system config HW (cpu arch, which GPU and compute capability), GPU device driver version, CUDA Toolkit version, OS flavor and version, PyTorch version, exact Triton nightly version installed (run 'pip show triton' to get the v3.0.0 suffix) as the min required info as part of repro recipe.

EDIT: I misspoke on one thing in the last paragraph. While the cmake issue I pointed out is valid. It's not the root cause (assuming steps in readme were followed). There is a bug in build script that would cause linux builds to default to x86-64 (if not aarch64 target) llvm build even if user is building for another arch on Linux and provided env variables pointing to their local build. I put up a PR for that #3719.

Thanks. My build machine is the typical x86_64 with Ubuntu 20.04.6 LTS, not SPARC. Maybe something is wrong with the MCJIT of llvm-project. I tried this building steps, it works normally now.