hipcc fails when cuda is installed in /usr

[sthibaul]

Hello,

When cuda is installed in /usr, hipcc in cuda mode fails to compile anything:

$ echo $CUDA_PATH
$ cat test.hip
void f(void) {
}
$ hipcc -v --x cu -c test.hip -o test.o
#$ _NVVM_BRANCH_=nvvm
#$ _SPACE_= 
#$ _CUDART_=cudart
#$ _HERE_=/usr/lib/nvidia-cuda-toolkit/bin
#$ _THERE_=/usr/lib/nvidia-cuda-toolkit/bin
#$ _TARGET_SIZE_=
#$ _TARGET_DIR_=
#$ _TARGET_SIZE_=64
#$ NVVMIR_LIBRARY_DIR=/usr/lib/nvidia-cuda-toolkit/libdevice
#$ PATH=/usr/lib/nvidia-cuda-toolkit/bin:/usr/bin:/bin:/opt/rocm/hip/bin
#$ LIBRARIES=  -L/usr/lib/x86_64-linux-gnu/stubs -L/usr/lib/x86_64-linux-gnu
#$ gcc -D__CUDA_ARCH__=520 -E -x c++  -DCUDA_DOUBLE_MATH_FUNCTIONS -D__CUDACC__ -D__NVCC__  -isystem "/usr/include" -isystem "/net/inria/soft/x86_64/rocm-5.1.0/hip/include"  -D__CUDACC_VER_MAJOR__=11 -D__CUDACC_VER_MINOR__=4 -D__CUDACC_VER_BUILD__=152 -D__CUDA_API_VER_MAJOR__=11 -D__CUDA_API_VER_MINOR__=4 -include "cuda_runtime.h" -m64 "test.hip" -o "/tmp/tmpxft_002df6f5_00000000-7_test.cpp1.ii" 
In file included from /usr/include/crt/math_functions.h:8958,
                 from /usr/include/crt/common_functions.h:303,
                 from /usr/include/cuda_runtime.h:115,
                 from <command-line>:
/usr/include/c++/11/cmath:45:15: fatal error: math.h: Aucun fichier ou dossier de ce type
   45 | #include_next <math.h>
      |               ^~~~~~~~
compilation terminated.
# --error 0x1 --

The -isystem /usr/include is what seems to be spurious, dropping it fixes things, as the attached patch does: patch.txt

This can be easily reproduced by using a debian (buster) or ubuntu image with the nvidia-cuda-toolkit rocm-hip-runtime-dev packages installed and the $CUDA_PATH environment variable set to /usr

Samuel

[cgmb]

This problem also affects hipcc for the AMD platform. It had to be patched for Debian packaging, as HIP will be installed to /usr in that case. As far as I understand, it's never valid to pass -isystem <system include path>.

[cgmb]

There's a number of other install locations that would be equally problematic (e.g., /usr/local) and it would be difficult to extend your patch to handle them all. For that reason, I think it might be better to apply the same fix as was used in https://github.com/llvm/llvm-project/commit/6730b44480fcce18bfbbae0c46719250e9eae425.

@sthibaul, could you perhaps try replacing all uses of isystem with idirafter and giving hipcc a test drive? You can use sed -i.bak 's/isystem/idirafter/g' /opt/rocm/bin/hipcc.pl to quickly apply the change, and revert it with mv /opt/rocm/bin/hipcc.pl{.bak,} when you're done.

[sthibaul]

could you perhaps try replacing all uses of isystem with idirafter

I am getting

€ hipcc -v --x cu -c test.hip  -o test.o
nvcc fatal   : Unknown option '-idirafter'

[cgmb]

Opps. That's certainly not going to work. Thanks for checking!

[ppanchad-amd]

@sthibaul Apologies for the lack of response. Can you please test with latest ROCm 6.0.2 (HIP 6.0.32831)? If resolved, please close ticket. Thanks!

[sthibaul]

I only have ROCm 6.0.0 at hand, but it seems fixed there indeed