ROCm
hipcc disregards `--cuda-gpu-arch` from CMake when determining `default_amdgpu_target`
We've come across an issue of hipcc always adding system device architectures to executable binaries even when not asked. This increases compile times for unaware users and causes a hurdle for those extracting GPU binaries, as there may always be some bloat.
This happens because the CMake cache variable AMDGPU_TARGETS gets turned into GPU_TARGETS internally by hip-config.cmake for the hip::device target, which then turns into compiler and linker flags with the name --cuda-gpu-arch. However hipcc was not updated to accomodate the change caused by moving from --amd-gpu-target to --cuda-gpu-arch and only checks for --amdgpu-target on the command line when determining whether to set default_amdgpu_target perl variable to 0. This variable controls whether the user asked for a GPU arch explicitly or should some default be embedded into the executable. That default is obtained by invoking rocminfo for GPUs inside the system. Currently trying to control binary archs explicitly via AMDGPU_TARGETS (alone) is not possible, because all system GPU archs are embedded into the executable.
Please search for --cuda-gpu-arch too on the CLI when deciding on defaulting to anything. Definition of opt to look for is here and usage is here.
Hotfix: hipcc not only looks for --amdgpu-target but also the HCC_AMDGPU_TARGET environment variable to determine whether to embed default targets. Setting them both makes arch selection explicit enough for hipcc. (Note: AMDGPU_TARGETS is a list in CMake format, so it is a semi-colon delimited list (may require extra hoops and defensive quotes around cli args, depending on your shell), while the HCC_AMDGPU_TARGET env var is just colon delimited.)
Ran into this too when building rccl.
this has been fixed in ROCm 5.7
https://github.com/ROCm-Developer-Tools/clr/blob/rocm-5.7.x/hipamd/hip-config-amd.cmake#L121
Ran into this too when building rccl.
this has been fixed in ROCm 5.7
https://github.com/ROCm-Developer-Tools/clr/blob/rocm-5.7.x/hipamd/hip-config-amd.cmake#L121
Thank you, Sam.