HSA directive no longer supported

[syifan]

It seems in the recent version of ROCm compilers, any assembly directive that has "hsa" in it is not supported and the compiler is giving an error on unsupported directives. Those directives are heavily used in the examples of this repository. Is there an updated version of those directives?

I tried to remove the directives in the assembly. The assembly file can be compiled to binary. However, the host program cannot launch the kernels because they cannot find the symbols in the binary file.

[ghostplant]

I have got this issue, have you solved it?

Scanning dependencies of target asm-kernel
[ 33%] Building CXX object examples/asm-kernel/CMakeFiles/asm-kernel.dir/asm-kernel.cpp.o
[ 35%] Linking CXX executable asm-kernel
[ 35%] Built target asm-kernel
Scanning dependencies of target asm-kernel_co
[ 37%] Assembling asm-kernel.s to asm-kernel.o
/root/LLVM-AMDGPU-Assembler-Extra/examples/asm-kernel/asm-kernel.s:43:1: error: unknown directive
.hsa_code_object_version 2,0
^
/root/LLVM-AMDGPU-Assembler-Extra/examples/asm-kernel/asm-kernel.s:44:1: error: unknown directive
.hsa_code_object_isa 8, 0, 3, "AMD", "AMDGPU"
^
/root/LLVM-AMDGPU-Assembler-Extra/examples/asm-kernel/asm-kernel.s:48:1: error: unknown directive
.amdgpu_hsa_kernel hello_world
^
/root/LLVM-AMDGPU-Assembler-Extra/examples/asm-kernel/asm-kernel.s:52:4: error: unknown directive
   .amd_kernel_code_t
   ^
/root/LLVM-AMDGPU-Assembler-Extra/examples/asm-kernel/asm-kernel.s:61:3: error: unknown directive
  .end_amd_kernel_code_t
  ^
examples/asm-kernel/CMakeFiles/asm-kernel_co.dir/build.make:64: recipe for target 'examples/asm-kernel/asm-kernel.o' failed
make[2]: *** [examples/asm-kernel/asm-kernel.o] Error 1
CMakeFiles/Makefile2:429: recipe for target 'examples/asm-kernel/CMakeFiles/asm-kernel_co.dir/all' failed
make[1]: *** [examples/asm-kernel/CMakeFiles/asm-kernel_co.dir/all] Error 2
Makefile:140: recipe for target 'all' failed
make: *** [all] Error 2

[syifan]

@ghostplant No, still have exactly the same errors. What I eventually do is to write OpenCL kernel with inline assembly. But I think that is not a solution to this problem.

[atamazov]

Try giving the "-mno-code-object-v3" option to clang, please.

[ghostplant]

@atamazov Thanks, this option can fix that problem. However, seems like the assembly sources are no longer compatible with gfx906:

/root/LLVM-AMDGPU-Assembler-Extra/examples/gfx8/fp16_storage.s:78:3: error: instruction not supported on this GPU
  v_add_u32     v1, vcc, s0, v0
  ^
/root/LLVM-AMDGPU-Assembler-Extra/examples/gfx8/fp16_storage.s:80:3: error: instruction not supported on this GPU
  v_addc_u32    v2, vcc, v2, 0, vcc
  ^
...

Do AMDGPUs with different gcnArch have different assembly code compatibility?

[atamazov]

Yes, gfx8 and gfx9 ISA differ.

[ghostplant]

@atamazov Because asm sources for gfx8 is not the one I want, I dump the isa code by setting KMDUMPISA=1 for hcc/hipcc command, and now I get the isa source code dump-gfx906.isa for gfx906 like this:

    .text
    .hsa_code_object_version 2,1
    .hsa_code_object_isa 9,0,6,"AMD","AMDGPU"
    .weak   _Z13vector_squareIfEvPT_PKS0_i ; -- Begin function _Z13vector_squareIfEvPT_PKS0_i
    .p2align    8
    .type   _Z13vector_squareIfEvPT_PKS0_i,@function
    .amdgpu_hsa_kernel _Z13vector_squareIfEvPT_PKS0_i
_Z13vector_squareIfEvPT_PKS0_i:         ; @_Z13vector_squareIfEvPT_PKS0_i
    .amd_kernel_code_t
        amd_code_version_major = 1
        amd_code_version_minor = 2
        amd_machine_kind = 1
        amd_machine_version_major = 9
        amd_machine_version_minor = 0
        ...

How can I compile this file dump-gfx906.isa into dump-gfx906.isabin? (so that I can further make use of lld to turn isabin into hsaco)

[ghostplant]

Solved, I got the solution by using llvm-mc

[atamazov]

Solved, I got the solution by using llvm-mc

Please publish it here for others who might encounter the same problem.

[atamazov]

@atamazov Because asm sources for gfx8 is not the one I want, I dump the isa code by setting KMDUMPISA=1 for hcc/hipcc command, and now I get the isa source code dump-gfx906.isa for gfx906 like this...

Please attach dump-gfx906.isa here. I would like to look into it, thanks.

dump-gfx906.isa:196:18: error: .amd_amdgpu_isa directive does not match triple and/or mcpu arguments specified through the command line .amd_amdgpu_isa "amdgcn-amd-amdhsa--gfx906+sram-ecc"

Please look at this doc: https://llvm.org/docs/AMDGPUUsage.html. It explains some details related to code-object-v3 and sram-ecc.

[ghostplant]

llvm-mc -mattr=-code-object-v3 -triple amdgcn-amd-amdhsa -mcpu=gfx906 -filetype=obj llvm_code.isa -o llvm_code.isabin

[ghostplant]

@atamazov This is the corresponding file llvm_code.isa:

dump-gfx906.isa.txt

[atamazov]

Let's close this, https://github.com/ROCm-Developer-Tools/LLVM-AMDGPU-Assembler-Extra/issues/20#issuecomment-489243561