Building with ROCm/HIP fails on a system without GPU

[lahwaacz]

The cuda_lt.sh script contains a --offload-arch=native flag for amdclang:

https://github.com/openucx/ucc/blob/c1734db1b2bc9ffeba5d17b3e81e1a9425dee100/cuda_lt.sh#L31

This should select the native architecture of the GPU present in the build system. However, if the build system does not have any GPU, the command fails:

$ /opt/rocm/lib/llvm/bin/amdclang -c -x hip -target x86_64-unknown-linux-gnu --offload-arch=gfx908 --offload-arch=gfx90a --offload-arch=gfx940 --offload-arch=gfx941 --offload-arch=gfx942 --offload-arch=gfx1030 --offload-arch=gfx1100 --offload-arch=gfx1101 --offload-arch=gfx1102 --offload-arch=native ec_rocm_executor_kernel.cu -I/usr/include/ -D__HIP_PLATFORM_AMD__ -I/opt/rocm/include/hip -I/opt/rocm/include -I/opt/rocm/llvm/include -I/opt/rocm/include/hsa -I/opt/rocm/include -I/build/openucc/src/ucc-1.3.0 -I/build/openucc/src/ucc-1.3.0 -I/build/openucc/src/ucc-1.3.0/src -I/build/openucc/src/ucc-1.3.0/src -I/build/openucc/src/ucc-1.3.0/src/components/ec/rocm -fPIC -O3 -o ./.libs/ec_rocm_executor_kernel.o
/opt/rocm/lib/llvm/bin/amdclang -c -x hip -target x86_64-unknown-linux-gnu --offload-arch=gfx908 --offload-arch=gfx90a --offload-arch=gfx940 --offload-arch=gfx941 --offload-arch=gfx942 --offload-arch=gfx1030 --offload-arch=gfx1100 --offload-arch=gfx1101 --offload-arch=gfx1102 --offload-arch=native ec_rocm_reduce.cu -I/usr/include/ -D__HIP_PLATFORM_AMD__ -I/opt/rocm/include/hip -I/opt/rocm/include -I/opt/rocm/llvm/include -I/opt/rocm/include/hsa -I/opt/rocm/include -I/build/openucc/src/ucc-1.3.0 -I/build/openucc/src/ucc-1.3.0 -I/build/openucc/src/ucc-1.3.0/src -I/build/openucc/src/ucc-1.3.0/src -I/build/openucc/src/ucc-1.3.0/src/components/ec/rocm -fPIC -O3 -o ./.libs/ec_rocm_reduce.o
clang: error: cannot determine amdgcn architecture: /opt/rocm/lib/llvm/bin/amdgpu-arch: ; consider passing it via '--offload-arch'
clang: error: cannot determine amdgcn architecture: /opt/rocm/lib/llvm/bin/amdgpu-arch: ; consider passing it via '--offload-arch'

[edgargabriel]

@lahwaacz thank you for the bug report, we will look into this. The UCC CI checker runs through exactly the same scenario (i.e. compiling UCC with the ROCm stack installed but without an AMD GPU being available). However, because of an issue that we faced with clang-tidy newer than clang-12, we fixed the ROCm version in the UCC CI to ROCm 5.7.1 - which still uses hipcc to compile the kernels vs. the new clang --offload-arch=... approach that we use with ROCm > 6.0.

For now, I think your best options are either to compile on a platform with an AMD GPU present, or change the cuda_lt.sh file and remove the --offload-arch=native argument.

[lahwaacz]

@edgargabriel Thanks, I've patched it for the Arch Linux package: https://gitlab.archlinux.org/archlinux/packaging/packages/openucc/-/commit/f5618b46d08fa2c41f218366871d17133145cde9#9b9baac1eb9b72790eef5540a1685306fc43fd6c_50_42

[romintomasetti]

Hi @edgargabriel !

We also encountered the same issue (cannot determine amdgcn architecture), since we're building ucc from inside a docker build step (such that devices like GPUs are not exposed).

It would be nice that we can provide the list of architectures that we want to compile for at the configuration step. For now, there are many ROCm architectures listed in cuda_lt.sh, but we only need a few of them (not to mention those not listed). An option like --offload-arch=A,B,C would be welcome. It could default to what's already in cuda_lt.sh if not provided, for backward compatibility. The same remark can be made for the enabled CUDA architectures. It would be nice if only a chosen subset could be passed when compiling ucc (it would help us reduce the compile time and size).

Note that we circumvented the problem by patching cuda_lt.sh to remove the native offloading.

[edgargabriel]

@romintomasetti thank you, it is on our list, we definitely plan to have it fixed for the next release. I think the fix is not entirely trivial since cuda_lt.sh is used by both cuda and rocm component, so it might require a deeper rework of that part of the code section.

[romintomasetti]

Also, please note that there is already the option --with-nvcc-gencode for CUDA: https://github.com/openucx/ucc/blob/1522ccff5e107451d747b1085b3f84714a6c2eea/config/m4/cuda.m4#L109-L118

[lahwaacz]

Fixed in https://github.com/openucx/ucc/releases/tag/v1.4.4