HIP icon indicating copy to clipboard operation
HIP copied to clipboard

Published 20 hours ago • verified publisher icon ROCm

`hipcc` disregards the `--gcc-toolchain` option

Open fwyzard opened this issue 3 years ago • 3 comments

Comparing the behaviour of various builds of clang, it looks like hipcc and the version of clang coming with HIP fail to make use of the --gcc-toolchain option.

To better illustrate what I mean, here is a comparison of the behaviour of different builds of clang:

  • clang 14.0.0 from CentOS 8 Stream
  • clang 12.0.1 from a private build
  • clang 14.0.0 from HIP/ROCm 5.0.2
  • hipcc 5.0.13601 / AMD clang version 14.0.0 from HIP/ROCm 5.0.2

common

$ alias format="sed -e's/=/=\n  /g' -e's/:\//\n  \//g'"
$ TOOLCHAIN=/cvmfs/cms.cern.ch/el8_amd64_gcc12/external/gcc/12.1.1-bf4aef5069fdf6bb6f77f897bcc8a6ae
$ $TOOLCHAIN/bin/g++ --version
g++ (GCC) 12.1.1 20220516
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

clang 14.0.0 from CentOS 8 Stream

$ /usr/bin/clang++ --version
clang version 14.0.0 (Red Hat 14.0.0-1.module_el8.7.0+1142+5343df54)
Target: x86_64-redhat-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

$ /usr/bin/clang++ --print-search-dirs | format
programs: =
  /usr/bin
  /usr/bin/../lib/gcc/x86_64-redhat-linux/8/../../../../x86_64-redhat-linux/bin
libraries: =
  /usr/lib64/clang/14.0.0
  /usr/bin/../lib/gcc/x86_64-redhat-linux/8
  /usr/bin/../lib/gcc/x86_64-redhat-linux/8/../../../../lib64
  /lib/../lib64
  /usr/lib/../lib64
  /usr/bin/../lib
  /usr/bin/../lib64
  /lib
  /usr/lib

$ /usr/bin/clang++ --print-search-dirs --gcc-toolchain=$TOOLCHAIN | format
programs: =
  /usr/bin
  /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/gcc/12.1.1-bf4aef5069fdf6bb6f77f897bcc8a6ae/lib/gcc/x86_64-redhat-linux-gnu/12.1.1/../../../../x86_64-redhat-linux-gnu/bin
libraries: =
  /usr/lib64/clang/14.0.0
  /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/gcc/12.1.1-bf4aef5069fdf6bb6f77f897bcc8a6ae/lib/gcc/x86_64-redhat-linux-gnu/12.1.1
  /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/gcc/12.1.1-bf4aef5069fdf6bb6f77f897bcc8a6ae/lib/gcc/x86_64-redhat-linux-gnu/12.1.1/../../../../lib64
  /lib/../lib64
  /usr/lib/../lib64
  /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/gcc/12.1.1-bf4aef5069fdf6bb6f77f897bcc8a6ae/lib/gcc/x86_64-redhat-linux-gnu/12.1.1/../../../../x86_64-redhat-linux-gnu/lib
  /usr/bin/../lib
  /usr/bin/../lib64
  /lib
  /usr/lib

clang 12.0.1 from a private build

$ source /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/llvm/12.0.1-a100c412676d03d68415d6def72fe659/etc/profile.d/init.sh

$ /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/llvm/12.0.1-a100c412676d03d68415d6def72fe659/bin/clang++ --version
clang version 12.0.1
Target: x86_64-redhat-linux-gnu
Thread model: posix
InstalledDir: /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/llvm/12.0.1-a100c412676d03d68415d6def72fe659/bin

$ /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/llvm/12.0.1-a100c412676d03d68415d6def72fe659/bin/clang++ --print-search-dirs | format
programs: =
  /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/llvm/12.0.1-a100c412676d03d68415d6def72fe659/bin
  /usr/lib/gcc/x86_64-redhat-linux/8/../../../../x86_64-redhat-linux/bin
libraries: =
  /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/llvm/12.0.1-a100c412676d03d68415d6def72fe659/lib64/clang/12.0.1
  /usr/lib/gcc/x86_64-redhat-linux/8
  /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64
  /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/llvm/12.0.1-a100c412676d03d68415d6def72fe659/bin/../lib64
  /lib/../lib64
  /usr/lib/../lib64
  /usr/lib/gcc/x86_64-redhat-linux/8/../../..
  /lib
  /usr/lib

$ /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/llvm/12.0.1-a100c412676d03d68415d6def72fe659/bin/clang++ --print-search-dirs --gcc-toolchain=$TOOLCHAIN | format
programs: =
  /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/llvm/12.0.1-a100c412676d03d68415d6def72fe659/bin
  /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/gcc/12.1.1-bf4aef5069fdf6bb6f77f897bcc8a6ae/lib/gcc/x86_64-redhat-linux-gnu/12.1.1/../../../../x86_64-redhat-linux-gnu/bin
libraries: =
  /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/llvm/12.0.1-a100c412676d03d68415d6def72fe659/lib64/clang/12.0.1
  /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/gcc/12.1.1-bf4aef5069fdf6bb6f77f897bcc8a6ae/lib/gcc/x86_64-redhat-linux-gnu/12.1.1
  /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/gcc/12.1.1-bf4aef5069fdf6bb6f77f897bcc8a6ae/lib/gcc/x86_64-redhat-linux-gnu/12.1.1/../../../../lib64
  /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/llvm/12.0.1-a100c412676d03d68415d6def72fe659/bin/../lib64
  /lib/../lib64
  /usr/lib/../lib64
  /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/gcc/12.1.1-bf4aef5069fdf6bb6f77f897bcc8a6ae/lib/gcc/x86_64-redhat-linux-gnu/12.1.1/../../../../x86_64-redhat-linux-gnu/lib
  /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/gcc/12.1.1-bf4aef5069fdf6bb6f77f897bcc8a6ae/lib/gcc/x86_64-redhat-linux-gnu/12.1.1/../../..
  /lib
  /usr/lib

clang 14.0.0 from HIP/ROCm 5.0.2

$ /cvmfs/patatrack.cern.ch/externals/x86_64/rhel8/amd/rocm-5.0.2/llvm/bin/clang++ --version
AMD clang version 14.0.0 (https://github.com/RadeonOpenCompute/llvm-project roc-5.0.2 22065 030a405a181176f1a7749819092f4ef8ea5f0758)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /cvmfs/patatrack.cern.ch/externals/x86_64/rhel8/amd/rocm-5.0.2/llvm/bin

$ /cvmfs/patatrack.cern.ch/externals/x86_64/rhel8/amd/rocm-5.0.2/llvm/bin/clang++ --print-search-dirs | format
programs: =
  /cvmfs/patatrack.cern.ch/externals/x86_64/rhel8/amd/rocm-5.0.2/llvm/bin
  /usr/lib/gcc/x86_64-redhat-linux/8/../../../../x86_64-redhat-linux/bin
libraries: =
  /cvmfs/patatrack.cern.ch/externals/x86_64/rhel8/amd/rocm-5.0.2/llvm/lib/clang/14.0.0
  /usr/lib/gcc/x86_64-redhat-linux/8
  /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64
  /lib/../lib64
  /usr/lib/../lib64
  /cvmfs/patatrack.cern.ch/externals/x86_64/rhel8/amd/rocm-5.0.2/llvm/bin/../lib
  /lib
  /usr/lib

$ /cvmfs/patatrack.cern.ch/externals/x86_64/rhel8/amd/rocm-5.0.2/llvm/bin/clang++ --print-search-dirs --gcc-toolchain=$TOOLCHAIN | format
programs: =
  /cvmfs/patatrack.cern.ch/externals/x86_64/rhel8/amd/rocm-5.0.2/llvm/bin
libraries: =
  /cvmfs/patatrack.cern.ch/externals/x86_64/rhel8/amd/rocm-5.0.2/llvm/lib/clang/14.0.0
  /lib/../lib64
  /usr/lib/../lib64
  /cvmfs/patatrack.cern.ch/externals/x86_64/rhel8/amd/rocm-5.0.2/llvm/bin/../lib
  /lib
  /usr/lib

hipcc 5.0.13601 / AMD clang version 14.0.0 from HIP/ROCm 5.0.2

$ /cvmfs/patatrack.cern.ch/externals/x86_64/rhel8/amd/rocm-5.0.2/bin/hipcc --version
HIP version: 5.0.13601-ded05588
AMD clang version 14.0.0 (https://github.com/RadeonOpenCompute/llvm-project roc-5.0.2 22065 030a405a181176f1a7749819092f4ef8ea5f0758)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /cvmfs/patatrack.cern.ch/externals/x86_64/rhel8/amd/rocm-5.0.2/llvm/bin

$ /cvmfs/patatrack.cern.ch/externals/x86_64/rhel8/amd/rocm-5.0.2/bin/hipcc --print-search-dirs | format
programs: =
  /cvmfs/patatrack.cern.ch/externals/x86_64/rhel8/amd/rocm-5.0.2/llvm/bin
  /usr/lib/gcc/x86_64-redhat-linux/8/../../../../x86_64-redhat-linux/bin
libraries: =
  /cvmfs/patatrack.cern.ch/externals/x86_64/rhel8/amd/rocm-5.0.2/llvm/lib/clang/14.0.0
  /usr/lib/gcc/x86_64-redhat-linux/8
  /usr/lib/gcc/x86_64-redhat-linux/8/../../../../lib64
  /lib/../lib64
  /usr/lib/../lib64
  /cvmfs/patatrack.cern.ch/externals/x86_64/rhel8/amd/rocm-5.0.2/llvm/bin/../lib
  /lib
  /usr/lib

$ /cvmfs/patatrack.cern.ch/externals/x86_64/rhel8/amd/rocm-5.0.2/bin/hipcc --print-search-dirs --gcc-toolchain=$TOOLCHAIN | format
programs: =
  /cvmfs/patatrack.cern.ch/externals/x86_64/rhel8/amd/rocm-5.0.2/llvm/bin
libraries: =
  /cvmfs/patatrack.cern.ch/externals/x86_64/rhel8/amd/rocm-5.0.2/llvm/lib/clang/14.0.0
  /lib/../lib64
  /usr/lib/../lib64
  /cvmfs/patatrack.cern.ch/externals/x86_64/rhel8/amd/rocm-5.0.2/llvm/bin/../lib
  /lib
  /usr/lib

summary

With clang 14.0.0 from CentOS 8 Stream and clang 12.0.1 from a private build, --gcc-toolchain=TOOLCHAIN makes clang++ use TOOLCHAIN to find include files and libraries.

With clang 14.0.0 and hipcc 5.0.13601 from HIP/ROCm 5.0.2, --gcc-toolchain=TOOLCHAIN makes clang++ and hipcc drop some of the system paths, but it does not add TOOLCHAIN to the paths used to find include files and libraries.

Is there a way to use hipcc (and its clang++ counterpart) with an explicit GCC toolchain ?

Should the behaviour of the hipcc --gcc-toolchain=TOOLCHAIN be corrected to match that of the upstream build of clang++ ?

fwyzard avatar Aug 23 '22 22:08 fwyzard

I think I have found the reason.

hipcc (and the version of clang coming with it) default to the target x86_64-unknown-linux-gnu:

$ /cvmfs/patatrack.cern.ch/externals/x86_64/rhel8/amd/rocm-5.0.2/bin/hipcc --version | grep Target
Target: x86_64-unknown-linux-gnu

clang from CentOS and our local build use x86_64-redhat-linux-gnu

$ /usr/bin/clang++ --version | grep Target
Target: x86_64-redhat-linux-gnu

$ /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/llvm/12.0.1-a100c412676d03d68415d6def72fe659/bin/clang++ --version | grep Target
Target: x86_64-redhat-linux-gnu

And the local build of GCC uses x86_64-redhat-linux-gnu:

$ TOOLCHAIN=/cvmfs/cms.cern.ch/el8_amd64_gcc12/external/gcc/12.1.1-bf4aef5069fdf6bb6f77f897bcc8a6ae
$ ls $TOOLCHAIN | grep -e -
x86_64-redhat-linux-gnu

So hipcc does not find the target triple in the $TOOLCHAIN directory, and ignores it.

Passing explicitly the x86_64-redhat-linux-gnu target makes hipcc pick up the directories:

$ /cvmfs/patatrack.cern.ch/externals/x86_64/rhel8/amd/rocm-5.0.2/bin/hipcc --print-search-dirs --target=x86_64-redhat-linux-gnu --gcc-toolchain=$TOOLCHAIN | format
programs: =
  /cvmfs/patatrack.cern.ch/externals/x86_64/rhel8/amd/rocm-5.0.2/llvm/bin
  /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/gcc/12.1.1-bf4aef5069fdf6bb6f77f897bcc8a6ae/lib/gcc/x86_64-redhat-linux-gnu/12.1.1/../../../../x86_64-redhat-linux-gnu/bin
libraries: =
  /cvmfs/patatrack.cern.ch/externals/x86_64/rhel8/amd/rocm-5.0.2/llvm/lib/clang/14.0.0
  /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/gcc/12.1.1-bf4aef5069fdf6bb6f77f897bcc8a6ae/lib/gcc/x86_64-redhat-linux-gnu/12.1.1
  /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/gcc/12.1.1-bf4aef5069fdf6bb6f77f897bcc8a6ae/lib/gcc/x86_64-redhat-linux-gnu/12.1.1/../../../../lib64
  /lib/../lib64
  /usr/lib/../lib64
  /cvmfs/cms.cern.ch/el8_amd64_gcc12/external/gcc/12.1.1-bf4aef5069fdf6bb6f77f897bcc8a6ae/lib/gcc/x86_64-redhat-linux-gnu/12.1.1/../../../../x86_64-redhat-linux-gnu/lib
  /cvmfs/patatrack.cern.ch/externals/x86_64/rhel8/amd/rocm-5.0.2/llvm/bin/../lib
  /lib
  /usr/lib

fwyzard avatar Aug 24 '22 08:08 fwyzard

Just a suggestion: for the CentOS/RedHat builds, would it make sense to use the x86_64-redhat-linux-gnu target instead of x86_64-unknown-linux-gnu ?

fwyzard avatar Aug 24 '22 09:08 fwyzard

@fwyzard Hi were you able to resolve your issue on the latest HIP? If so, can we please close this ticket?

abhimeda avatar Feb 07 '24 18:02 abhimeda