'disable-symbolication' build error

[gdkrmr]

I get the following build error on Manjaro:

I added the rocm libraries from arch4edu and replaced libstdc++.so provided by Julia with the one provided by the system by copying it. I get the same error with the Julia 1.5.2 binaries and current Julia master.

I set up the environment as follows:

LD_LIBRARY_PATH="/opt/rocm/hsa/lib/"
LD_LIBRARY_PATH="/opt/rocm/lib/:$LD_LIBRARY_PATH"
PATH="/opt/rocm/llvm/bin/:$PATH"

   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.0-DEV.1046 (2020-09-25)
 _/ |\__'_|_|_|\__'_|  |  Commit 55aeb2ff01 (6 days old master)
|__/                   |

(@v1.6) pkg> add AMDGPU#master
...

(@v1.6) pkg> build AMDGPU
   Building AMDGPU → `~/.julia/packages/AMDGPU/ztzIl/deps/build.log`
ERROR: Error building `AMDGPU`: 
: CommandLine Error: Option 'disable-symbolication' registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options

[jpsamaroo]

This is a very unfortunate situation caused by issues with weak symbols causing linking against the wrong libLLVM: https://github.com/JuliaLang/julia/issues/36342. The core issue is that multiple users of LLVM (Julia and probably ROCm comgr) register options in LLVM on startup, which causes the same LLVM options to be registered more than once in the same library (hence the error).

The only way I know of to work around this is to rebuild your ROCm stack to use a static build of LLVM, which is then directly compiled into the ROCm stack components that use it. Unfortunately, LLVM.jl cannot be run with Julia built against static LLVM. The directions for how to compile a static LLVM are here: https://paste.sr.ht/~jpsamaroo/462a58b32961d93227120173e26d91cdb410cc1a. You then need to build ROCT and ROCR against this LLVM; let me know if you need help getting them to build and run.

[jpsamaroo]

Slight edit: Technically ROCR and ROCT don't depend on LLVM, although other components in the ROCm stack do. You should probably build ROCT and ROCR first, ensure those work in Julia, then get the rest of the stack (like rocBLAS, rocFFT, etc.) built with your static LLVM build once you get the chance.

[gdkrmr]

will this get fixed by #6?

[jpsamaroo]

Yes, that would be a valid solution to this problem for uses not relying on external libraries (BLAS, FFT, etc.).

[antholzer]

@jpsamaroo did you get rocBLAS to work that way? As soon as I install it (or put it into LD_LIBRARY_PATH) I get the 'disable-symbolication' build error and I did build it using a statically compiled LLVM. rocFFT and rocRAND do not cause this.

[jpsamaroo]

TBH I haven't built them locally, and I think they might not be running on CI either (not sure why). @vchuravy any ideas why this would happen with our deps being statically linked to LLVM?

[vchuravy]

Static global variables aliasing each other... probably.

[jpsamaroo]

Wonderful. Maybe we could solve this by shipping rocBLAS (and the other libs) as artifacts, linked against Julia's LLVM?

[gdkrmr]

I haven't gotten compute to run at all on the gpu, it seems that the Ryzen 4750U gpu is not supported by rocm (https://github.com/RadeonOpenCompute/ROCm/issues/1101). It seems that the closed source driver do support it and I may try it.

[jpsamaroo]

Some basic parts of AMDGPU do work on my Renoir APU (I can allocate a ROCArray and do some regular kernel launches), but running the tests causes my display manager to hang, so I've not been doing much of that :smile:

[gdkrmr]

@jpsamaroo What OS and drivers are you using? Running clinfo causes my laptop to freeze :-(

[jpsamaroo]

Alpine Linux (musl) with a patched ROCT on 3.8 and ROCR on 3.5, currently using the ROCK kernel for my laptop and 5.9 RC on my server (which runs AMDGPU CI). I don't have rocBLAS, rocFFT, et. al installed since build times are horrid and Alpine doesn't have packages for anything ROCm-related. rocm_smi.py works fine for me.

[jpsamaroo]

@gdkrmr #187 should improve this situation significantly, by shipping artifacts built against a static LLVM. Please give it a try (and make sure to run a current Linux kernel). I have my Renoir APU passing tests with it :slightly_smiling_face:

[sofiageo]

Hey, I somehow stumbled upon this issue and I thought it would be a good idea to check Julia AMDGPU on my system. I tried to run it wasn't working at first, but the stacktrace helped me and after a long investigation I found out that there was something wrong with a library path (rocalution) of my ROCm 5.0 installation. After fixing that I get normal output from JuliaGPU.

HSA Runtime (ready)
- Path: /home/sofiageo/.julia/artifacts/b1aa837f69ba67b20f9654af56c818e6d9bfd262/lib/libhsa-runtime64.so
- Version: 1.1.0
- Initialized: true
ld.lld (ready)
- Path: /home/sofiageo/.julia/artifacts/836082a6a472e357ce62db92868086df6735be27/bin/lld
ROCm-Device-Libs (ready)
- Path: /home/sofiageo/.julia/artifacts/84cbee8fa15e34e61aeb7d3c17f24c8c0d3214e1/amdgcn/bitcode
- Downloaded: false
HIP Runtime (ready)
- Path: /home/sofiageo/.julia/artifacts/b5a35fe56035e3d95e3203689c38aafec324a861/hip/lib/libamdhip64.so
rocBLAS (ready)
- Path: /opt/rocm-5.0.0/lib/librocblas.so
rocSOLVER (ready)
- Path: /opt/rocm-5.0.0/lib/librocsolver.so
rocFFT (ready)
- Path: /opt/rocm-5.0.0/lib/librocfft.so
rocRAND (ready)
- Path: /home/sofiageo/.julia/artifacts/3c565b6e353eb2ec9345ac55ba0f2d6be3444bea/hiprand/lib/../../rocrand/lib/librocrand.so.1
rocSPARSE (ready)
- Path: /opt/rocm-5.0.0/lib/librocsparse.so
rocALUTION (ready)
- Path: /opt/rocm/lib/librocalution.so
MIOpen (ready)
- Path: /opt/rocm/lib/libMIOpen.so
HSA Agents (2):
- GPU: Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] (gfx1010)
- CPU: AMD Ryzen 7 2700X Eight-Core Processor

I didn't want to open a new issue just to say it works for me so I hope I'm not creating much spam here :)