HIP clang vs. deprecated HCC

[tdd11235813]

Can you shed some light on the new compiler workflow on HIP(AMD)? We currently use the deprecated HCC for our HIP codes on the AMD platform (ROCm 2.5). By reading hipcc script, it looks like, the new workflow will involve the /opt/rocm/llvm as default for HIP clang path, which is currently not present on our ROCm 2.5 installation (ubuntu), but it probably will come with new ROCm? Or is it part of the llvm AMDGPU toolchain?

[tdd11235813]

update, WIP: working on compiling hip-clang via INSTALL.md (please fix AMD-clang link to HIP-clang in TOC). Here is the current install script. It compiles, but could not compile hip code with hip-clang yet. It still uses HCC stuff, but it seems it is intended. I'll update when I know more...

manual install hip-clang (broken)

#!/bin/bash
# https://github.com/ROCm-Developer-Tools/HIP/blob/master/INSTALL.md#hip-clang

# requirements:
# libelf-dev and rocm libs with hcc (atm still required for compiling HIP)

#BUILD_TYPE="RelWithDebInfo"
BUILD_TYPE="Release"

SRC_DIR=$HOME/sources/hipclang
LLVM_INSTALL_DIR=/opt/rocm/llvm # make sure you have the privileges
HIP_INSTALL_DIR=${LLVM_INSTALL_DIR}/hip # within llvm dir to avoid clash with regular hip installation

mkdir -p $SRC_DIR && cd $SRC_DIR

# Users need to build LLVM, clang, lld, ROCm device library, and HIP from source
# llvm
[ -d "llvm/" ] || git clone --single-branch --recursive -b amd-common https://github.com/RadeonOpenCompute/llvm.git llvm

cd ${SRC_DIR}/llvm/tools
[ -d "clang/" ] || git clone --single-branch --recursive -b amd-common https://github.com/RadeonOpenCompute/clang
[ -d "lld/" ] || git clone --single-branch --recursive -b amd-common https://github.com/RadeonOpenCompute/lld

# build llvm/clang/lld
cd ${SRC_DIR}/llvm
if [ ! -d "build/" ]; then
    mkdir build && cd build
    cmake .. \
          -DLLVM_BUILD_TESTS=OFF \
          -DLLVM_PARALLEL_COMPILE_JOBS=12 \
          -DLLVM_PARALLEL_LINK_JOBS=12 \
          -DLLVM_BUILD_UTILS=OFF \
          -DLLVM_TARGETS_TO_BUILD="AMDGPU;X86" \
          -DCMAKE_BUILD_TYPE=${BUILD_TYPE} \
          -DCMAKE_INSTALL_PREFIX=${LLVM_INSTALL_DIR} \
          -DLLVM_ENABLE_TERMINFO=OFF # libncurses issue, https://github.com/android-ndk/ndk/issues/574#issuecomment-373580520
    make -j 12 || exit 1
    make install || exit 1
fi



# build ROCm device library with clang built in last branch
cd ${SRC_DIR}
[ -d "rocmdevlib/" ] || git clone --single-branch --recursive -b master https://github.com/RadeonOpenCompute/ROCm-Device-Libs.git rocmdevlib
cd rocmdevlib

if [ ! -d "build/" ]; then
    mkdir build && cd build

    CC=${LLVM_INSTALL_DIR}/bin/clang \
      cmake .. \
      -DCMAKE_BUILD_TYPE=${BUILD_TYPE} \
      -DLLVM_DIR=${LLVM_INSTALL_DIR}
    make -j 12 || exit 1
fi


# build HIP (with HCC?) from installed ROCm packages
cd $SRC_DIR
[ -d "HIP/" ] || git clone --single-branch --recursive -b master https://github.com/ROCm-Developer-Tools/HIP.git
cd HIP

if [ ! -d "build/" ]; then
    mkdir -p build && cd build
    export DEVICE_LIB_PATH=${SRC_DIR}/rocmdevlib/build
    cmake .. \
          -DCMAKE_BUILD_TYPE=${BUILD_TYPE} \
          -DHIP_COMPILER=clang \
          -DCMAKE_INSTALL_PREFIX=${HIP_INSTALL_DIR} \
          -DHSA_PATH=/opt/rocm/hsa \
          -DHCC_HOME=/opt/rocm/hcc
    make -j 12 || exit 1
    make install || exit 1
fi

# After installation, make sure HIP_PATH is pointed to /where/to/install/hip.

# By default HIP looks for HSA in /opt/rocm/hsa (can be overridden by setting HSA_PATH environment variable)
# By default HIP is installed into /opt/rocm/hip (can be overridden by setting HIP_PATH environment variable).
# By default HIP looks for clang in /opt/rocm/llvm/bin (can be overridden by setting HIP_CLANG_PATH environment variable)
# By default HIP looks for device library in /opt/rocm/lib (can be overriden by setting DEVICE_LIB_PATH environment variable).
# Optionally, consider adding /opt/rocm/bin to your PATH to make it easier to use the tools.
# Optionally, set HIPCC_VERBOSE=7 to output the command line for compilation to make sure clang is used instead of hcc.

echo
echo "Build was successfull."
echo
echo "Compile your code with HIP-clang by using the following environment variables"
echo
echo "export HIP_PATH=${HIP_INSTALL_DIR}"
echo "export HIP_CLANG_PATH=${LLVM_INSTALL_DIR}/bin"
echo "export DEVICE_LIB_PATH=${SRC_DIR}/rocmdevlib/build" # using source dir for now

update: the new hip-clang will not use any HCC at all, so do not waste your time to debug this script. The instructions in the public repos are not consistent at the moment.

[davidrohr]

I tried this recipe, but when I try to compile a hip file, I get:

/usr/lib/gcc/x86_64-pc-linux-gnu/8.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: /tmp/a-a5df7e.o: in function `__hip_module_ctor':
a.hip.cpp:(.text+0x706): undefined reference to `__hipRegisterFatBinary'
/usr/lib/gcc/x86_64-pc-linux-gnu/8.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: a.hip.cpp:(.text+0x739): undefined reference to `__hipRegisterFunction'
/usr/lib/gcc/x86_64-pc-linux-gnu/8.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: /tmp/a-a5df7e.o: in function `__hip_module_dtor':
a.hip.cpp:(.text+0x75e): undefined reference to `__hipUnregisterFatBinary'

This is defined in hip_clang.cpp, and this file is also compiled by the above script. Really waiting when hip-clang will be part of a ROCm release.

[gargrahul]

@davidrohr Could you please try with #1331 ?

[davidrohr]

Thanks, the fix works.

[davidrohr]

I found another issue with the hip-clang from this recipe: When the command line invoking hipcc contains -fopenmp, I receive the following errors:

/opt/rocm/llvm/lib/clang/10.0.0/include/__clang_cuda_math_forward_declares.h:53:16: error: static declaration of 'abs' follows non-static declaration
__DEVICE__ int abs(int) __NOEXCEPT;
               ^
/usr/include/stdlib.h:837:12: note: previous declaration is here
extern int abs (int __x) __THROW __attribute__ ((__const__)) __wur;
           ^
In file included from a.hip.cpp:3:
In file included from /opt/rocm/llvm/hip//include/hip/hip_runtime.h:56:
In file included from /opt/rocm/llvm/hip//include/hip/hcc_detail/hip_runtime.h:457:

Please not that I am not compiling for cuda, I have no idea why hipcc pull in the clang cuda headers.

[tdd11235813]

thanks @davidrohr for pushing this, I also had issues with OpenMP and hcc in the past, not only because of the fishy math includes, and had to disable hipcc and OpenMP. I know it does not help :( (we use HIP for nvcc and hcc backend at the moment, without OpenMP support)

There are several people wondering about this HIP-clang path. It might save a lot of our time if AMD could provide a first HIP-clang version or straightforward guide or at least some information on the current plan & issues, since HCC has been deprecated (have not checked recent updates though).

When an HCC-less HIP-clang will be provided eventually, then it would be now waste of time to debug this HIP-clang compilation where HCC is still used.

[tdd11235813]

update: the new hip-clang will not use any HCC at all, so do not waste your time to debug this script. The instructions in the public repos are not consistent at the moment.

[lamikr]

Can you give some information which branch of llvm and hip could be used for building the hip-clang version so that I could build and install it by myself?

I have been able to build older hcc based hip and use that for building the tensorflow 2.02 from AMD's tensorflow-upsteam repo's r2.0-rocm-hipclang branch thats using the hip clang version. (same problem with the r2.1 branches)

I have only been able to build the tensorflow if I change the flags on r2.0-rocm-hipclang branch to use the hcc based hip.

[ppanchad-amd]

@lamikr Apologies for the lack of response. Can you please test with latest ROCm 6.1.0 (HIP 6.1)? Thanks!