Installing TE is so hard.

[ZihaoZheng98]

Meeting problems such as

File "/TransformerEngine/build_tools/build_ext.py", line 91, in _build_cmake raise RuntimeError(f"Error when running CMake: {e}") RuntimeError: Error when running CMake: Command '['/usr/local/lib/python3.10/dist-packages/cmake/data/bin/cmake', '-S', '/TransformerEngine/transformer_engine/common', '-B', '/TransformerEngine/build/cmake', '-DPython_EXECUTABLE=/usr/bin/python3.10', '-DPython_INCLUDE_DIR=/usr/include/python3.10', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_INSTALL_PREFIX=/TransformerEngine/build/lib.linux-x86_64-3.10', '-DCMAKE_CUDA_ARCHITECTURES=70;80;89;90', '-Dpybind11_DIR=/usr/local/lib/python3.10/dist-packages/pybind11/share/cmake/pybind11', '-GNinja']' returned non-zero exit status 1.

and ninja -v

[timmoon10]

https://github.com/NVIDIA/TransformerEngine/issues/355#issuecomment-2394353816

[wplf]

The real problem shows above.

[ZihaoZheng98]

Thanks, I will have a try. @wplf @timmoon10

[klhhhhh]

I agreed!!! 😮‍💨

[saurabh-kataria]

Following works for me after trying several combinations:

export TMPDIR=/tmp && export CMAKE_TEMP_DIR=/tmp && export BUILD_DIR=/tmp/build
mkdir -p $TMPDIR && mkdir -p $CMAKE_TEMP_DIR && mkdir -p $BUILD_DIR
TMP_DIR=$TMPDIR MAX_JOBS=1 CUDA_HOME=$CUDA_HOME CUDNN_PATH=$CUDNN_PATH CC=$CC CXX=$CXX pip -v install --cache-dir /tmp/pip-cache --no-deps git+https://github.com/NVIDIA/TransformerEngine.git@stable --no-deps

where your env vars are something like below

export CC=$GCCPATH/bin/gcc
export CXX=$GCCPATH/bin/g++
export CUDA_HOME=/home/$USER/cuda-12.3
export CUDNN_PATH=/home/$USER/cudnn-9.2.0/usr

[mertunsall]

same issue!

[mertunsall]

Using pip 25.0 from /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages/pip (python 3.10)
Non-user install because site-packages writeable
Created temporary directory: /tmp/pip-build-tracker-viieds1s
Initialized build tracking at /tmp/pip-build-tracker-viieds1s
Created build tracker: /tmp/pip-build-tracker-viieds1s
Entered build tracker: /tmp/pip-build-tracker-viieds1s
Created temporary directory: /tmp/pip-install-jojazlk_
Created temporary directory: /tmp/pip-ephem-wheel-cache-8yb1nntq
Processing /home/mertunsal/TransformerEngine
  Added file:///home/mertunsal/TransformerEngine to build tracker '/tmp/pip-build-tracker-viieds1s'
  Running setup.py (path:/home/mertunsal/TransformerEngine/setup.py) egg_info for package from file:///home/mertunsal/TransformerEngine
  Created temporary directory: /tmp/pip-pip-egg-info-cq8jyf1u
  Running command python setup.py egg_info
  WARNING: Skipping transformer_engine_cu12 as it is not installed.
  WARNING: Skipping transformer_engine_torch as it is not installed.
  WARNING: Skipping transformer_engine_jax as it is not installed.
  running egg_info
  creating /tmp/pip-pip-egg-info-cq8jyf1u/transformer_engine.egg-info
  writing /tmp/pip-pip-egg-info-cq8jyf1u/transformer_engine.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-pip-egg-info-cq8jyf1u/transformer_engine.egg-info/dependency_links.txt
  writing requirements to /tmp/pip-pip-egg-info-cq8jyf1u/transformer_engine.egg-info/requires.txt
  writing top-level names to /tmp/pip-pip-egg-info-cq8jyf1u/transformer_engine.egg-info/top_level.txt
  writing manifest file '/tmp/pip-pip-egg-info-cq8jyf1u/transformer_engine.egg-info/SOURCES.txt'
  reading manifest file '/tmp/pip-pip-egg-info-cq8jyf1u/transformer_engine.egg-info/SOURCES.txt'
  adding license file 'LICENSE'
  writing manifest file '/tmp/pip-pip-egg-info-cq8jyf1u/transformer_engine.egg-info/SOURCES.txt'
  Preparing metadata (setup.py) ... done
  Source in /home/mertunsal/TransformerEngine has version 2.3.0.dev0+31f5c2d, which satisfies requirement transformer_engine==2.3.0.dev0+31f5c2d from file:///home/mertunsal/TransformerEngine
  Removed transformer_engine==2.3.0.dev0+31f5c2d from file:///home/mertunsal/TransformerEngine from build tracker '/tmp/pip-build-tracker-viieds1s'
Requirement already satisfied: torch>=2.1 in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from transformer_engine==2.3.0.dev0+31f5c2d) (2.6.0)
Requirement already satisfied: importlib-metadata>=1.0 in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from transformer_engine==2.3.0.dev0+31f5c2d) (8.6.1)
Requirement already satisfied: packaging in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from transformer_engine==2.3.0.dev0+31f5c2d) (24.2)
Requirement already satisfied: pydantic in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from transformer_engine==2.3.0.dev0+31f5c2d) (2.11.1)
Requirement already satisfied: zipp>=3.20 in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from importlib-metadata>=1.0->transformer_engine==2.3.0.dev0+31f5c2d) (3.21.0)
Requirement already satisfied: filelock in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from torch>=2.1->transformer_engine==2.3.0.dev0+31f5c2d) (3.18.0)
Requirement already satisfied: typing-extensions>=4.10.0 in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from torch>=2.1->transformer_engine==2.3.0.dev0+31f5c2d) (4.12.2)
Requirement already satisfied: networkx in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from torch>=2.1->transformer_engine==2.3.0.dev0+31f5c2d) (3.3)
Requirement already satisfied: jinja2 in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from torch>=2.1->transformer_engine==2.3.0.dev0+31f5c2d) (3.1.6)
Requirement already satisfied: fsspec in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from torch>=2.1->transformer_engine==2.3.0.dev0+31f5c2d) (2024.6.1)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.4.127 in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from torch>=2.1->transformer_engine==2.3.0.dev0+31f5c2d) (12.4.127)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.4.127 in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from torch>=2.1->transformer_engine==2.3.0.dev0+31f5c2d) (12.4.127)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.4.127 in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from torch>=2.1->transformer_engine==2.3.0.dev0+31f5c2d) (12.4.127)
Requirement already satisfied: nvidia-cudnn-cu12==9.1.0.70 in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from torch>=2.1->transformer_engine==2.3.0.dev0+31f5c2d) (9.1.0.70)
Requirement already satisfied: nvidia-cublas-cu12==12.4.5.8 in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from torch>=2.1->transformer_engine==2.3.0.dev0+31f5c2d) (12.4.5.8)
Requirement already satisfied: nvidia-cufft-cu12==11.2.1.3 in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from torch>=2.1->transformer_engine==2.3.0.dev0+31f5c2d) (11.2.1.3)
Requirement already satisfied: nvidia-curand-cu12==10.3.5.147 in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from torch>=2.1->transformer_engine==2.3.0.dev0+31f5c2d) (10.3.5.147)
Requirement already satisfied: nvidia-cusolver-cu12==11.6.1.9 in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from torch>=2.1->transformer_engine==2.3.0.dev0+31f5c2d) (11.6.1.9)
Requirement already satisfied: nvidia-cusparse-cu12==12.3.1.170 in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from torch>=2.1->transformer_engine==2.3.0.dev0+31f5c2d) (12.3.1.170)
Requirement already satisfied: nvidia-cusparselt-cu12==0.6.2 in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from torch>=2.1->transformer_engine==2.3.0.dev0+31f5c2d) (0.6.2)
Requirement already satisfied: nvidia-nccl-cu12==2.21.5 in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from torch>=2.1->transformer_engine==2.3.0.dev0+31f5c2d) (2.21.5)
Requirement already satisfied: nvidia-nvtx-cu12==12.4.127 in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from torch>=2.1->transformer_engine==2.3.0.dev0+31f5c2d) (12.4.127)
Requirement already satisfied: nvidia-nvjitlink-cu12==12.4.127 in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from torch>=2.1->transformer_engine==2.3.0.dev0+31f5c2d) (12.4.127)
Requirement already satisfied: triton==3.2.0 in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from torch>=2.1->transformer_engine==2.3.0.dev0+31f5c2d) (3.2.0)
Requirement already satisfied: sympy==1.13.1 in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from torch>=2.1->transformer_engine==2.3.0.dev0+31f5c2d) (1.13.1)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from sympy==1.13.1->torch>=2.1->transformer_engine==2.3.0.dev0+31f5c2d) (1.3.0)
Requirement already satisfied: annotated-types>=0.6.0 in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from pydantic->transformer_engine==2.3.0.dev0+31f5c2d) (0.7.0)
Requirement already satisfied: pydantic-core==2.33.0 in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from pydantic->transformer_engine==2.3.0.dev0+31f5c2d) (2.33.0)
Requirement already satisfied: typing-inspection>=0.4.0 in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from pydantic->transformer_engine==2.3.0.dev0+31f5c2d) (0.4.0)
Requirement already satisfied: MarkupSafe>=2.0 in /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages (from jinja2->torch>=2.1->transformer_engine==2.3.0.dev0+31f5c2d) (2.1.5)
Created temporary directory: /tmp/pip-unpack-sjij7jnk
Building wheels for collected packages: transformer_engine
  Created temporary directory: /tmp/pip-wheel-fn60s7yv
  Building wheel for transformer_engine (setup.py) ...   Destination directory: /tmp/pip-wheel-fn60s7yv
  Running command python setup.py bdist_wheel
  WARNING: Skipping transformer_engine_cu12 as it is not installed.
  WARNING: Skipping transformer_engine_torch as it is not installed.
  WARNING: Skipping transformer_engine_jax as it is not installed.
  running bdist_wheel
  running build
  running build_py
  creating build/lib.linux-x86_64-cpython-310/transformer_engine
  copying transformer_engine/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine
  creating build/lib.linux-x86_64-cpython-310/transformer_engine/common
  copying transformer_engine/common/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/common
  copying transformer_engine/common/utils.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/common
  creating build/lib.linux-x86_64-cpython-310/transformer_engine/jax
  copying transformer_engine/jax/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax
  copying transformer_engine/jax/activation.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax
  copying transformer_engine/jax/attention.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax
  copying transformer_engine/jax/dense.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax
  copying transformer_engine/jax/layernorm.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax
  copying transformer_engine/jax/layernorm_dense.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax
  copying transformer_engine/jax/layernorm_mlp.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax
  copying transformer_engine/jax/setup.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax
  copying transformer_engine/jax/sharding.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax
  copying transformer_engine/jax/softmax.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax
  creating build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
  copying transformer_engine/pytorch/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
  copying transformer_engine/pytorch/attention.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
  copying transformer_engine/pytorch/constants.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
  copying transformer_engine/pytorch/cpu_offload.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
  copying transformer_engine/pytorch/cross_entropy.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
  copying transformer_engine/pytorch/distributed.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
  copying transformer_engine/pytorch/float8_tensor.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
  copying transformer_engine/pytorch/fp8.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
  copying transformer_engine/pytorch/graph.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
  copying transformer_engine/pytorch/jit.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
  copying transformer_engine/pytorch/numerics_debug.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
  copying transformer_engine/pytorch/permutation.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
  copying transformer_engine/pytorch/setup.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
  copying transformer_engine/pytorch/softmax.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
  copying transformer_engine/pytorch/transformer.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
  copying transformer_engine/pytorch/utils.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch
  creating build/lib.linux-x86_64-cpython-310/transformer_engine/common/recipe
  copying transformer_engine/common/recipe/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/common/recipe
  creating build/lib.linux-x86_64-cpython-310/transformer_engine/jax/cpp_extensions
  copying transformer_engine/jax/cpp_extensions/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax/cpp_extensions
  copying transformer_engine/jax/cpp_extensions/activation.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax/cpp_extensions
  copying transformer_engine/jax/cpp_extensions/attention.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax/cpp_extensions
  copying transformer_engine/jax/cpp_extensions/base.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax/cpp_extensions
  copying transformer_engine/jax/cpp_extensions/gemm.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax/cpp_extensions
  copying transformer_engine/jax/cpp_extensions/misc.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax/cpp_extensions
  copying transformer_engine/jax/cpp_extensions/normalization.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax/cpp_extensions
  copying transformer_engine/jax/cpp_extensions/quantization.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax/cpp_extensions
  copying transformer_engine/jax/cpp_extensions/softmax.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax/cpp_extensions
  creating build/lib.linux-x86_64-cpython-310/transformer_engine/jax/flax
  copying transformer_engine/jax/flax/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax/flax
  copying transformer_engine/jax/flax/module.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax/flax
  copying transformer_engine/jax/flax/transformer.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax/flax
  creating build/lib.linux-x86_64-cpython-310/transformer_engine/jax/praxis
  copying transformer_engine/jax/praxis/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax/praxis
  copying transformer_engine/jax/praxis/module.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax/praxis
  copying transformer_engine/jax/praxis/transformer.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax/praxis
  creating build/lib.linux-x86_64-cpython-310/transformer_engine/jax/quantize
  copying transformer_engine/jax/quantize/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax/quantize
  copying transformer_engine/jax/quantize/dequantizer.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax/quantize
  copying transformer_engine/jax/quantize/helper.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax/quantize
  copying transformer_engine/jax/quantize/metadata.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax/quantize
  copying transformer_engine/jax/quantize/quantizer.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax/quantize
  copying transformer_engine/jax/quantize/scaling_modes.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax/quantize
  copying transformer_engine/jax/quantize/tensor.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/jax/quantize
  creating build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/cpp_extensions
  copying transformer_engine/pytorch/cpp_extensions/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/cpp_extensions
  copying transformer_engine/pytorch/cpp_extensions/fused_attn.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/cpp_extensions
  copying transformer_engine/pytorch/cpp_extensions/gemm.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/cpp_extensions
  creating build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/dot_product_attention
  copying transformer_engine/pytorch/dot_product_attention/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/dot_product_attention
  copying transformer_engine/pytorch/dot_product_attention/inference.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/dot_product_attention
  copying transformer_engine/pytorch/dot_product_attention/rope.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/dot_product_attention
  copying transformer_engine/pytorch/dot_product_attention/utils.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/dot_product_attention
  creating build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/module
  copying transformer_engine/pytorch/module/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/module
  copying transformer_engine/pytorch/module/_common.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/module
  copying transformer_engine/pytorch/module/base.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/module
  copying transformer_engine/pytorch/module/fp8_padding.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/module
  copying transformer_engine/pytorch/module/fp8_unpadding.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/module
  copying transformer_engine/pytorch/module/grouped_linear.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/module
  copying transformer_engine/pytorch/module/layernorm.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/module
  copying transformer_engine/pytorch/module/layernorm_linear.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/module
  copying transformer_engine/pytorch/module/layernorm_mlp.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/module
  copying transformer_engine/pytorch/module/linear.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/module
  copying transformer_engine/pytorch/module/rmsnorm.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/module
  creating build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops
  copying transformer_engine/pytorch/ops/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops
  copying transformer_engine/pytorch/ops/_common.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops
  copying transformer_engine/pytorch/ops/fuser.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops
  copying transformer_engine/pytorch/ops/linear.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops
  copying transformer_engine/pytorch/ops/op.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops
  copying transformer_engine/pytorch/ops/sequential.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops
  creating build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/optimizers
  copying transformer_engine/pytorch/optimizers/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/optimizers
  copying transformer_engine/pytorch/optimizers/fused_adam.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/optimizers
  copying transformer_engine/pytorch/optimizers/fused_sgd.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/optimizers
  copying transformer_engine/pytorch/optimizers/multi_tensor_apply.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/optimizers
  creating build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/tensor
  copying transformer_engine/pytorch/tensor/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/tensor
  copying transformer_engine/pytorch/tensor/float8_tensor.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/tensor
  copying transformer_engine/pytorch/tensor/mxfp8_tensor.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/tensor
  copying transformer_engine/pytorch/tensor/quantized_tensor.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/tensor
  copying transformer_engine/pytorch/tensor/utils.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/tensor
  creating build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/triton
  copying transformer_engine/pytorch/triton/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/triton
  copying transformer_engine/pytorch/triton/cross_entropy.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/triton
  copying transformer_engine/pytorch/triton/permutation.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/triton
  creating build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops/basic
  copying transformer_engine/pytorch/ops/basic/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops/basic
  copying transformer_engine/pytorch/ops/basic/activation.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops/basic
  copying transformer_engine/pytorch/ops/basic/add_in_place.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops/basic
  copying transformer_engine/pytorch/ops/basic/all_gather.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops/basic
  copying transformer_engine/pytorch/ops/basic/all_reduce.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops/basic
  copying transformer_engine/pytorch/ops/basic/basic_linear.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops/basic
  copying transformer_engine/pytorch/ops/basic/bias.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops/basic
  copying transformer_engine/pytorch/ops/basic/identity.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops/basic
  copying transformer_engine/pytorch/ops/basic/layer_norm.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops/basic
  copying transformer_engine/pytorch/ops/basic/make_extra_output.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops/basic
  copying transformer_engine/pytorch/ops/basic/quantize.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops/basic
  copying transformer_engine/pytorch/ops/basic/reduce_scatter.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops/basic
  copying transformer_engine/pytorch/ops/basic/reshape.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops/basic
  copying transformer_engine/pytorch/ops/basic/rmsnorm.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops/basic
  creating build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops/fused
  copying transformer_engine/pytorch/ops/fused/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops/fused
  copying transformer_engine/pytorch/ops/fused/backward_linear_add.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops/fused
  copying transformer_engine/pytorch/ops/fused/forward_linear_bias_activation.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops/fused
  copying transformer_engine/pytorch/ops/fused/forward_linear_bias_add.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops/fused
  copying transformer_engine/pytorch/ops/fused/userbuffers_backward_linear.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops/fused
  copying transformer_engine/pytorch/ops/fused/userbuffers_forward_linear.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/ops/fused
  creating build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/tensor/_internal
  copying transformer_engine/pytorch/tensor/_internal/__init__.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/tensor/_internal
  copying transformer_engine/pytorch/tensor/_internal/float8_tensor_base.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/tensor/_internal
  copying transformer_engine/pytorch/tensor/_internal/mxfp8_tensor_base.py -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/tensor/_internal
  running egg_info
  writing transformer_engine.egg-info/PKG-INFO
  writing dependency_links to transformer_engine.egg-info/dependency_links.txt
  writing requirements to transformer_engine.egg-info/requires.txt
  writing top-level names to transformer_engine.egg-info/top_level.txt
  reading manifest file 'transformer_engine.egg-info/SOURCES.txt'
  adding license file 'LICENSE'
  writing manifest file 'transformer_engine.egg-info/SOURCES.txt'
  /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages/setuptools/command/build_py.py:212: _Warning: Package 'transformer_engine.pytorch.csrc' is absent from the `packages` configuration.
  !!

          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'transformer_engine.pytorch.csrc' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.

          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'transformer_engine.pytorch.csrc' is explicitly added
          to the `packages` configuration field.

          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).

          You can read more about "package discovery" on setuptools documentation page:

          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html

          If you don't want 'transformer_engine.pytorch.csrc' to be distributed and are
          already explicitly excluding 'transformer_engine.pytorch.csrc' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.

          You can read more about "package data files" on setuptools documentation page:

          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html


          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************

  !!
    check.warn(importable)
  /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages/setuptools/command/build_py.py:212: _Warning: Package 'transformer_engine.pytorch.csrc.extensions' is absent from the `packages` configuration.
  !!

          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'transformer_engine.pytorch.csrc.extensions' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.

          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'transformer_engine.pytorch.csrc.extensions' is explicitly added
          to the `packages` configuration field.

          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).

          You can read more about "package discovery" on setuptools documentation page:

          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html

          If you don't want 'transformer_engine.pytorch.csrc.extensions' to be distributed and are
          already explicitly excluding 'transformer_engine.pytorch.csrc.extensions' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.

          You can read more about "package data files" on setuptools documentation page:

          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html


          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************

  !!
    check.warn(importable)
  /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages/setuptools/command/build_py.py:212: _Warning: Package 'transformer_engine.pytorch.csrc.extensions.multi_tensor' is absent from the `packages` configuration.
  !!

          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'transformer_engine.pytorch.csrc.extensions.multi_tensor' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.

          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'transformer_engine.pytorch.csrc.extensions.multi_tensor' is explicitly added
          to the `packages` configuration field.

          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).

          You can read more about "package discovery" on setuptools documentation page:

          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html

          If you don't want 'transformer_engine.pytorch.csrc.extensions.multi_tensor' to be distributed and are
          already explicitly excluding 'transformer_engine.pytorch.csrc.extensions.multi_tensor' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.

          You can read more about "package data files" on setuptools documentation page:

          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html


          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************

  !!
    check.warn(importable)
  creating build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/csrc
  copying transformer_engine/pytorch/csrc/common.cpp -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/csrc
  creating build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/csrc/extensions
  copying transformer_engine/pytorch/csrc/extensions/activation.cpp -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/csrc/extensions
  copying transformer_engine/pytorch/csrc/extensions/apply_rope.cpp -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/csrc/extensions
  copying transformer_engine/pytorch/csrc/extensions/attention.cu -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/csrc/extensions
  copying transformer_engine/pytorch/csrc/extensions/bias.cpp -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/csrc/extensions
  copying transformer_engine/pytorch/csrc/extensions/cast.cpp -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/csrc/extensions
  copying transformer_engine/pytorch/csrc/extensions/comm_gemm_overlap.cpp -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/csrc/extensions
  copying transformer_engine/pytorch/csrc/extensions/gemm.cpp -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/csrc/extensions
  copying transformer_engine/pytorch/csrc/extensions/misc.cpp -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/csrc/extensions
  copying transformer_engine/pytorch/csrc/extensions/normalization.cpp -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/csrc/extensions
  copying transformer_engine/pytorch/csrc/extensions/padding.cpp -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/csrc/extensions
  copying transformer_engine/pytorch/csrc/extensions/permutation.cu -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/csrc/extensions
  copying transformer_engine/pytorch/csrc/extensions/pybind.cpp -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/csrc/extensions
  copying transformer_engine/pytorch/csrc/extensions/quantizer.cpp -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/csrc/extensions
  copying transformer_engine/pytorch/csrc/extensions/recipe.cpp -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/csrc/extensions
  copying transformer_engine/pytorch/csrc/extensions/softmax.cpp -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/csrc/extensions
  copying transformer_engine/pytorch/csrc/extensions/swizzle.cpp -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/csrc/extensions
  copying transformer_engine/pytorch/csrc/extensions/transpose.cpp -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/csrc/extensions
  copying transformer_engine/pytorch/csrc/extensions/type_converters.cpp -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/csrc/extensions
  copying transformer_engine/pytorch/csrc/extensions/util.cpp -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/csrc/extensions
  creating build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/csrc/extensions/multi_tensor
  copying transformer_engine/pytorch/csrc/extensions/multi_tensor/multi_tensor_adam.cu -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/csrc/extensions/multi_tensor
  copying transformer_engine/pytorch/csrc/extensions/multi_tensor/multi_tensor_compute_scale.cu -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/csrc/extensions/multi_tensor
  copying transformer_engine/pytorch/csrc/extensions/multi_tensor/multi_tensor_l2norm_kernel.cu -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/csrc/extensions/multi_tensor
  copying transformer_engine/pytorch/csrc/extensions/multi_tensor/multi_tensor_scale_kernel.cu -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/csrc/extensions/multi_tensor
  copying transformer_engine/pytorch/csrc/extensions/multi_tensor/multi_tensor_sgd_kernel.cu -> build/lib.linux-x86_64-cpython-310/transformer_engine/pytorch/csrc/extensions/multi_tensor
  running build_ext
  Building CMake extension transformer_engine
  Running command /home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages/cmake/data/bin/cmake -S /home/mertunsal/TransformerEngine/transformer_engine/common -B /home/mertunsal/TransformerEngine/build/cmake -DPython_EXECUTABLE=/home/mertunsal/miniconda3/envs/verl/bin/python -DPython_INCLUDE_DIR=/home/mertunsal/miniconda3/envs/verl/include/python3.10 -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/home/mertunsal/TransformerEngine/build/lib.linux-x86_64-cpython-310 -DCMAKE_CUDA_ARCHITECTURES=70;80;89;90 -Dpybind11_DIR=/home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages/pybind11/share/cmake/pybind11 -GNinja
  CMake Error at /home/mertunsal/TransformerEngine/3rdparty/cudnn-frontend/cmake/cuDNN.cmake:3 (find_path):
    Could not find CUDNN_INCLUDE_DIR using the following files: cudnn.h
  Call Stack (most recent call first):
    CMakeLists.txt:44 (include)


  -- Configuring incomplete, errors occurred!
  Traceback (most recent call last):
    File "/home/mertunsal/TransformerEngine/build_tools/build_ext.py", line 89, in _build_cmake
      subprocess.run(command, cwd=build_dir, check=True)
    File "/home/mertunsal/miniconda3/envs/verl/lib/python3.10/subprocess.py", line 524, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['/home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages/cmake/data/bin/cmake', '-S', '/home/mertunsal/TransformerEngine/transformer_engine/common', '-B', '/home/mertunsal/TransformerEngine/build/cmake', '-DPython_EXECUTABLE=/home/mertunsal/miniconda3/envs/verl/bin/python', '-DPython_INCLUDE_DIR=/home/mertunsal/miniconda3/envs/verl/include/python3.10', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_INSTALL_PREFIX=/home/mertunsal/TransformerEngine/build/lib.linux-x86_64-cpython-310', '-DCMAKE_CUDA_ARCHITECTURES=70;80;89;90', '-Dpybind11_DIR=/home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages/pybind11/share/cmake/pybind11', '-GNinja']' returned non-zero exit status 1.

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "<string>", line 2, in <module>
    File "<pip-setuptools-caller>", line 34, in <module>
    File "/home/mertunsal/TransformerEngine/setup.py", line 173, in <module>
      setuptools.setup(
    File "/home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages/setuptools/__init__.py", line 117, in setup
      return distutils.core.setup(**attrs)
    File "/home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 186, in setup
      return run_commands(dist)
    File "/home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 202, in run_commands
      dist.run_commands()
    File "/home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 983, in run_commands
      self.run_command(cmd)
    File "/home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages/setuptools/dist.py", line 999, in run_command
      super().run_command(command)
    File "/home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 1002, in run_command
      cmd_obj.run()
    File "/home/mertunsal/TransformerEngine/setup.py", line 53, in run
      super().run()
    File "/home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages/setuptools/command/bdist_wheel.py", line 379, in run
      self.run_command("build")
    File "/home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 339, in run_command
      self.distribution.run_command(command)
    File "/home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages/setuptools/dist.py", line 999, in run_command
      super().run_command(command)
    File "/home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 1002, in run_command
      cmd_obj.run()
    File "/home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 136, in run
      self.run_command(cmd_name)
    File "/home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 339, in run_command
      self.distribution.run_command(command)
    File "/home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages/setuptools/dist.py", line 999, in run_command
      super().run_command(command)
    File "/home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 1002, in run_command
      cmd_obj.run()
    File "/home/mertunsal/TransformerEngine/build_tools/build_ext.py", line 119, in run
      ext._build_cmake(
    File "/home/mertunsal/TransformerEngine/build_tools/build_ext.py", line 91, in _build_cmake
      raise RuntimeError(f"Error when running CMake: {e}")
  RuntimeError: Error when running CMake: Command '['/home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages/cmake/data/bin/cmake', '-S', '/home/mertunsal/TransformerEngine/transformer_engine/common', '-B', '/home/mertunsal/TransformerEngine/build/cmake', '-DPython_EXECUTABLE=/home/mertunsal/miniconda3/envs/verl/bin/python', '-DPython_INCLUDE_DIR=/home/mertunsal/miniconda3/envs/verl/include/python3.10', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_INSTALL_PREFIX=/home/mertunsal/TransformerEngine/build/lib.linux-x86_64-cpython-310', '-DCMAKE_CUDA_ARCHITECTURES=70;80;89;90', '-Dpybind11_DIR=/home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages/pybind11/share/cmake/pybind11', '-GNinja']' returned non-zero exit status 1.
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /home/mertunsal/miniconda3/envs/verl/bin/python -u -c '
  exec(compile('"'"''"'"''"'"'
  # This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
  #
  # - It imports setuptools before invoking setup.py, to enable projects that directly
  #   import from `distutils.core` to work with newer packaging standards.
  # - It provides a clear error message when setuptools is not installed.
  # - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
  #   setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
  #     manifest_maker: standard file '"'"'-c'"'"' not found".
  # - It generates a shim setup.py, for handling setup.cfg-only projects.
  import os, sys, tokenize
  
  try:
      import setuptools
  except ImportError as error:
      print(
          "ERROR: Can not execute `setup.py` since setuptools is not available in "
          "the build environment.",
          file=sys.stderr,
      )
      sys.exit(1)
  
  __file__ = %r
  sys.argv[0] = __file__
  
  if os.path.exists(__file__):
      filename = __file__
      with tokenize.open(__file__) as f:
          setup_py_code = f.read()
  else:
      filename = "<auto-generated setuptools caller>"
      setup_py_code = "from setuptools import setup; setup()"
  
  exec(compile(setup_py_code, filename, "exec"))
  '"'"''"'"''"'"' % ('"'"'/home/mertunsal/TransformerEngine/setup.py'"'"',), "<pip-setuptools-caller>", "exec"))' bdist_wheel -d /tmp/pip-wheel-fn60s7yv
  cwd: /home/mertunsal/TransformerEngine/
error
  ERROR: Failed building wheel for transformer_engine
  Running setup.py clean for transformer_engine
  Running command python setup.py clean
  WARNING: Skipping transformer_engine_cu12 as it is not installed.
  WARNING: Skipping transformer_engine_torch as it is not installed.
  WARNING: Skipping transformer_engine_jax as it is not installed.
  running clean
  removing 'build/lib.linux-x86_64-cpython-310' (and everything under it)
  'build/bdist.linux-x86_64' does not exist -- can't clean it
  'build/scripts-3.10' does not exist -- can't clean it
Failed to build transformer_engine
Remote version of pip: 25.0.1
Local version of pip:  25.0
Was pip installed by pip? False
ERROR: Failed to build installable wheels for some pyproject.toml based projects (transformer_engine)
Exception information:
Traceback (most recent call last):
  File "/home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 106, in _run_wrapper
    status = _inner_run()
  File "/home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 97, in _inner_run
    return self.run(options, args)
  File "/home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 67, in wrapper
    return func(self, options, args)
  File "/home/mertunsal/miniconda3/envs/verl/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 435, in run
    raise InstallationError(
pip._internal.exceptions.InstallationError: Failed to build installable wheels for some pyproject.toml based projects (transformer_engine)
Removed build tracker: '/tmp/pip-build-tracker-viieds1s'