Add support for AMD target feature specifications in --offload-arch

[rbberger]

Using address sanitizer for ROCm seems to require to explicitly set xnack+ in the --offload-arch flag during compilation so that the necessary instrumentation is generated.

Explicitly use xnack+ in the offload architecture option. For example, --offload-arch=gfx90a:xnack+ Other architectures are allowed, but their device code will not be instrumented and a warning will be emitted.

Currently Kokkos doesn't provide a mechanism to manipulate that flag. One workaround is enabling CMake language support and taking control of that flag via CMAKE_HIP_ARCHITECTURES. However, the default CMake setup in Kokkos only lets you set architectures via the Kokkos_ARCH_AMD_<arch> Boolean options.

One proposal is to add an optional Kokkos_ARCH_AMD_TARGET_FEATURES CMake option that gets appended when set. An alternative so to provide an override to replace the entire --offload-arch=foo command-line option.

See also:

• List of target features in LLVM AMDGPU backend

[Rombur]

I don't think that providing something like Kokkos_ARCH_AMD_TARGET_FEATURES would work very well. It's fine for :xnack+ where the user could write Kokkos_ARCH_AMD_TARGET_FEATURES=":xnack+". It doesn't really work if you want to set -mtgsplit because now you need to write Kokkos_ARCH_AMD_TARGET_FEATURES=" -mtgsplit". Notice that you need a blank space before the flag. I've never seen anything like that before. On top of that, the order of the flags would matter. This ":xnack+ -mtgsplit" would work but this " -mtgsplit :xnack+" would not. I think it would be less error-prone to let the user override the entire --offload-arch=foo. We can provide such option but you will still need to set Kokkos_ARCH_AMD_<arch> (or let Kokkos detect it) and we won't do any sanity check. If the flag that you provide doesn't make sense or if you set Kokkos_ARCH_AMD_GFX90A and --offload-arch=gfx908, we won't try to catch the problem.