llvm icon indicating copy to clipboard operation
llvm copied to clipboard
verified publisher icon intel

LLVM with AMD backend fails to link binary object

Open jgtong opened this issue 3 years ago • 16 comments

Describe the bug LLVM compiler with AMD backend cannot link finally binary

To Reproduce Please describe the steps to reproduce the behavior:

  1. Include code snippet as short as possible: N/A

  2. Specify the command which should be used to compile the program:

The final clang++ command for linking the binary that caused the failure was this:

/home/jaytong/sycl_workspace/llvm/build/bin/clang++ -v  -O3 -fsycl  -fsycl-targets=amdgcn-amd-amdhsa -Xsycl-target-backend --offload-arch=gfx908   -Wall -Wno-unknown-pragmas -Wextra -stdlib=libstdc++ -fcolor-diagnostics -Qunused-arguments -O2 -g -DNDEBUG   CMakeFiles/ethminer.dir/main.cpp.o  -o ethminer   -L/home/jaytong/DPA_Projects/LIBS/json/1.9.5/lib/cmake/jsoncpp/../../../lib  -Wl,-rpath,/home/jaytong/DPA_Projects/LIBS/json/1.9.5/lib/cmake/jsoncpp/../../../lib:/home/jaytong/DPA_Projects/LIBS/boost/1.73.0/lib:/home/jaytong/DPA_Projects/LIBS/openssl/1.1.1f/lib: ../libethcore/libethcore.a ../libpoolprotocols/libpoolprotocols.a ../libdevcore/libdevcore.a libethminer-buildinfo.a /home/jaytong/DPA_Projects/LIBS/boost/1.73.0/lib/libboost_system.so.1.71.0 /home/jaytong/DPA_Projects/LIBS/boost/1.73.0/lib/libboost_thread.so.1.71.0 ../libapicore/libapicore.a -ljsoncpp /home/jaytong/DPA_Projects/LIBS/openssl/1.1.1f/lib/libssl.so /home/jaytong/DPA_Projects/LIBS/openssl/1.1.1f/lib/libcrypto.so ../libethcore/libethcore.a ../libethash-sycl/libethash-sycl.a ../libethcore/libethcore.a ../libethash-sycl/libethash-sycl.a /home/jaytong/DPA_Projects/LIBS/ethash/0.4.3/lib/libethash.a ../libhwmon/libhwmon.a -ldl ../libdevcore/libdevcore.a /home/jaytong/DPA_Projects/LIBS/boost/1.73.0/lib/libboost_system.so.1.71.0 -lpthread libethminer-buildinfo.a /home/jaytong/DPA_Projects/LIBS/boost/1.73.0/lib/libboost_filesystem.so.1.71.0

After specifying the -v in the clang++ command, the llvm-foreach reported an error after trying to do the following:

 "/home/jaytong/sycl_workspace/llvm/build/bin/llvm-foreach" --out-ext=out --in-file-list=/tmp/main-8dbbcf/main-gfx908.o --in-replace=/tmp/main-8dbbcf/main-gfx908.o --out-file-list=/tmp/main-f00a16/main-gfx908.out --out-replace=/tmp/main-f00a16/main-gfx908.out -- /home/jaytong/sycl_workspace/llvm/build/bin/clang-offload-bundler -unbundle -type=a -input=/home/jaytong/DPA_Projects/LIBS/json/1.9.5/lib/cmake/jsoncpp/../../../lib/libjsoncpp.a -targets=sycl-amdgcn-amd-amdhsa-gfx908 -output=/tmp/libbc-jsoncpp-amdgcn-gfx908-168cbe.a -allow-missing-bundles -hip-openmp-compatible
llvm-foreach: Couldn't find replace string for output in the command.
clang-15: error: amdgcn-link command failed with exit code 1 (use -v to see invocation)

  1. Specify the comment which should be used to launch the program: N/A

  2. Indicate what is wrong and what was expected: It should link the binary

Environment (please complete the following information):

  • OS: Ubuntu 20.04.4

  • Target device and vendor: AMD MI100 gfx908

  • DPC++ version: clang version 15.0.0 (https://github.com/intel/llvm 9749efa845abce0cbdad531ce540b7d0804cd868) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /home/jaytong/sycl_workspace/llvm/build/bin

  • Dependencies version: ROCM 5.1.0 SDK

jgtong avatar May 02 '22 21:05 jgtong

This may be related to #5996

zjin-lcf avatar May 05 '22 14:05 zjin-lcf

@AerialMantis Could you please flag the issue as "blocking" for developers and researchers ? Thanks.

zjin-lcf avatar May 06 '22 02:05 zjin-lcf

I also see a similar issue at commit 92cfd5365ae1040f4b530c6fa73cca7968951b7a:

clang -cc1 version 15.0.0 based upon LLVM 15.0.0git default target x86_64-unknown-linux-gnu
 "/workdir/amd/dpcpp/oss/llvm-intel/build/install/bin/llvm-foreach" --out-ext=out --in-file-list=/tmp/1-49c73a/1-gfx908.o --in-replace=/tmp/1-49c73a/1-gfx908.o --out-file-list=/tmp/1-0a53bb/1-gfx908.out --out-replace=/tmp/1-0a53bb/1-gfx908.out -- lld -flavor gnu --no-undefined -shared -plugin-opt=-amdgpu-internalize-symbols -plugin-opt=-sycl-enable-local-accessor -plugin-opt=mcpu=gfx908 -plugin-opt=O3 -o /tmp/1-0a53bb/1-gfx908.out /tmp/1-49c73a/1-gfx908.o
llvm-foreach: No such file or directory
clang-15: error: amdgcn-link command failed with exit code 1 (use -v to see invocation)

Compile command:

clang++ -fsycl -v -O3 -fsycl-targets=amdgcn-amd-amdhsa -Xsycl-target-backend --offload-arch=gfx908 1.cpp

It feels like /tmp/1-49c73a/1-gfx908.o file couldn't be found, which causes the issue. Though I'm not sure why it wasn't generated.

densamoilov avatar May 08 '22 21:05 densamoilov

Is there any update to this problem? I am blocked by this issue with a workload that I am trying to compile on the AMD platforms.

jgtong avatar May 18 '22 22:05 jgtong

@zjin-lcf, is there any update on this?

densamoilov avatar Jun 02 '22 03:06 densamoilov

Sorry, we need help.

zjin-lcf avatar Jun 02 '22 10:06 zjin-lcf

I found the reason for the issue I described here. The problem was that a path to LLD was not in PATH therefore the default linker was used causing the issue. Another question is why is LLD not installed in build/installed/bin directory but stays in build/bin?

densamoilov avatar Jun 22 '22 02:06 densamoilov

I found the reason for the issue I described here. The problem was that a path to LLD was not in PATH therefore the default linker was used causing the issue. Another question is why is LLD not installed in build/installed/bin directory but stays in build/bin?

Thank you for identifying this problem. Is there an environment variable that we need to set? Currently I have $PATH and $LD_LIBRARY_PATH set to the build/bin and build/lib respectively

jgtong avatar Jun 22 '22 02:06 jgtong

I added build/bin path to PATH variable and it helped. Per my understanding, lld command should work so that linking step can work as expected for AMD target.

densamoilov avatar Jun 22 '22 02:06 densamoilov

I added build/bin path to PATH variable and it helped. Per my understanding, lld command should work so that linking step can work as expected for AMD target.

Interesting, I already have $PATH in build/bin and did a which lld is already pointing to the llvm directory. Still not working for me syclos_amd/20220608/sycl_workspace/llvm/build/bin/lld

jgtong avatar Jun 22 '22 03:06 jgtong

jfyi, I used 49f72fa9ee6ac500e7f8277fa5ccabff88aaff2b

densamoilov avatar Jun 22 '22 05:06 densamoilov

jfyi, I used 49f72fa

@densamoilov

I tried your suggestion again using the hash commit, rebuilt the llvm compiler and included the build/bin/ and build/install/bin as part of my path. I am still seeing this error: llvm-foreach: Couldn't find replace string for output in the command

Here's the verbose output when trying to link the binary. I hope someone can help me resolve this issue? It is currently blocking one of my critical projects.

clang version 15.0.0 (https://github.com/intel/llvm 49f72fa9ee6ac500e7f8277fa5ccabff88aaff2b)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/jaytong/sycl_workspace/llvm/build/install/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7.5.0
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/8
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Candidate multilib: x32;@mx32
Selected multilib: .;@m64
Found CUDA installation: /usr/local/cuda, version 11.5
Found HIP installation: /opt/rocm, version 4.4.21401
clang-offload-bundler -type=ao -targets=host-x86_64-unknown-linux-gnu -input=../libethcore/libethcore.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr-intel-unknown -input=../libethcore/libethcore.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocx-intel-unknown -input=../libethcore/libethcore.a -check-section 
clang-offload-bundler -type=o -targets=sycl-spir64-unknown-unknown -input=CMakeFiles/ethminer.dir/main.cpp.o -check-section 
clang-offload-bundler -type=ao -targets=sycl-spir64-unknown-unknown -input=../libethcore/libethcore.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-spir64-unknown-unknown -input=../libpoolprotocols/libpoolprotocols.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-spir64-unknown-unknown -input=../libdevcore/libdevcore.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-spir64-unknown-unknown -input=libethminer-buildinfo.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-spir64-unknown-unknown -input=../libapicore/libapicore.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-spir64-unknown-unknown -input=../libethcore/libethcore.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-spir64-unknown-unknown -input=../libethash-sycl/libethash-sycl.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-spir64-unknown-unknown -input=../libethcore/libethcore.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-spir64-unknown-unknown -input=../libethash-sycl/libethash-sycl.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-spir64-unknown-unknown -input=/home/jaytong/DPA_Projects/LIBS/ethash/0.4.3/lib/libethash.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-spir64-unknown-unknown -input=../libhwmon/libhwmon.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-spir64-unknown-unknown -input=../libdevcore/libdevcore.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-spir64-unknown-unknown -input=libethminer-buildinfo.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocx-intel-unknown -input=../libethcore/libethcore.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr-intel-unknown -input=../libethcore/libethcore.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr_emu-intel-unknown -input=../libethcore/libethcore.a -check-section 
clang-offload-bundler -type=ao -targets=host-x86_64-unknown-linux-gnu -input=../libethcore/libethcore.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocx-intel-unknown -input=../libpoolprotocols/libpoolprotocols.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr-intel-unknown -input=../libpoolprotocols/libpoolprotocols.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr_emu-intel-unknown -input=../libpoolprotocols/libpoolprotocols.a -check-section 
clang-offload-bundler -type=ao -targets=host-x86_64-unknown-linux-gnu -input=../libpoolprotocols/libpoolprotocols.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocx-intel-unknown -input=../libdevcore/libdevcore.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr-intel-unknown -input=../libdevcore/libdevcore.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr_emu-intel-unknown -input=../libdevcore/libdevcore.a -check-section 
clang-offload-bundler -type=ao -targets=host-x86_64-unknown-linux-gnu -input=../libdevcore/libdevcore.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocx-intel-unknown -input=libethminer-buildinfo.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr-intel-unknown -input=libethminer-buildinfo.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr_emu-intel-unknown -input=libethminer-buildinfo.a -check-section 
clang-offload-bundler -type=ao -targets=host-x86_64-unknown-linux-gnu -input=libethminer-buildinfo.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocx-intel-unknown -input=../libapicore/libapicore.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr-intel-unknown -input=../libapicore/libapicore.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr_emu-intel-unknown -input=../libapicore/libapicore.a -check-section 
clang-offload-bundler -type=ao -targets=host-x86_64-unknown-linux-gnu -input=../libapicore/libapicore.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocx-intel-unknown -input=../libethcore/libethcore.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr-intel-unknown -input=../libethcore/libethcore.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr_emu-intel-unknown -input=../libethcore/libethcore.a -check-section 
clang-offload-bundler -type=ao -targets=host-x86_64-unknown-linux-gnu -input=../libethcore/libethcore.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocx-intel-unknown -input=../libethash-sycl/libethash-sycl.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr-intel-unknown -input=../libethash-sycl/libethash-sycl.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr_emu-intel-unknown -input=../libethash-sycl/libethash-sycl.a -check-section 
clang-offload-bundler -type=ao -targets=host-x86_64-unknown-linux-gnu -input=../libethash-sycl/libethash-sycl.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocx-intel-unknown -input=../libethcore/libethcore.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr-intel-unknown -input=../libethcore/libethcore.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr_emu-intel-unknown -input=../libethcore/libethcore.a -check-section 
clang-offload-bundler -type=ao -targets=host-x86_64-unknown-linux-gnu -input=../libethcore/libethcore.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocx-intel-unknown -input=../libethash-sycl/libethash-sycl.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr-intel-unknown -input=../libethash-sycl/libethash-sycl.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr_emu-intel-unknown -input=../libethash-sycl/libethash-sycl.a -check-section 
clang-offload-bundler -type=ao -targets=host-x86_64-unknown-linux-gnu -input=../libethash-sycl/libethash-sycl.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocx-intel-unknown -input=/home/jaytong/DPA_Projects/LIBS/ethash/0.4.3/lib/libethash.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr-intel-unknown -input=/home/jaytong/DPA_Projects/LIBS/ethash/0.4.3/lib/libethash.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr_emu-intel-unknown -input=/home/jaytong/DPA_Projects/LIBS/ethash/0.4.3/lib/libethash.a -check-section 
clang-offload-bundler -type=ao -targets=host-x86_64-unknown-linux-gnu -input=/home/jaytong/DPA_Projects/LIBS/ethash/0.4.3/lib/libethash.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocx-intel-unknown -input=../libhwmon/libhwmon.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr-intel-unknown -input=../libhwmon/libhwmon.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr_emu-intel-unknown -input=../libhwmon/libhwmon.a -check-section 
clang-offload-bundler -type=ao -targets=host-x86_64-unknown-linux-gnu -input=../libhwmon/libhwmon.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocx-intel-unknown -input=../libdevcore/libdevcore.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr-intel-unknown -input=../libdevcore/libdevcore.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr_emu-intel-unknown -input=../libdevcore/libdevcore.a -check-section 
clang-offload-bundler -type=ao -targets=host-x86_64-unknown-linux-gnu -input=../libdevcore/libdevcore.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocx-intel-unknown -input=libethminer-buildinfo.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr-intel-unknown -input=libethminer-buildinfo.a -check-section 
clang-offload-bundler -type=ao -targets=sycl-fpga_aocr_emu-intel-unknown -input=libethminer-buildinfo.a -check-section 
clang-offload-bundler -type=ao -targets=host-x86_64-unknown-linux-gnu -input=libethminer-buildinfo.a -check-section 
 "/home/jaytong/sycl_workspace/llvm/build/install/bin/clang-offload-bundler" -type=o -targets=host-x86_64-unknown-linux-gnu,sycl-amdgcn-amd-amdhsa-gfx900 -input=CMakeFiles/ethminer.dir/main.cpp.o -output=/tmp/main-5e8dac.o -output=/tmp/main-c3b415/main-gfx900.o -unbundle -allow-missing-bundles
 "/usr/bin/ld" -z relro --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o /tmp/main-f470e0.out /lib/x86_64-linux-gnu/crt1.o /lib/x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/9/crtbegin.o -L/home/jaytong/DPA_Projects/LIBS/json/1.9.5/lib/cmake/jsoncpp/../../../lib -L/usr/lib/gcc/x86_64-linux-gnu/9 -L/usr/lib/gcc/x86_64-linux-gnu/9/../../../../lib64 -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib64 -L/home/jaytong/sycl_workspace/llvm/build/install/bin/../lib -L/lib -L/usr/lib /tmp/main-5e8dac.o -rpath /home/jaytong/DPA_Projects/LIBS/json/1.9.5/lib/cmake/jsoncpp/../../../lib:/home/jaytong/DPA_Projects/LIBS/boost/1.71.0/lib:/home/jaytong/DPA_Projects/LIBS/openssl/1.1.1f/lib: ../libethcore/libethcore.a ../libpoolprotocols/libpoolprotocols.a ../libdevcore/libdevcore.a libethminer-buildinfo.a /home/jaytong/DPA_Projects/LIBS/boost/1.71.0/lib/libboost_system.so.1.71.0 /home/jaytong/DPA_Projects/LIBS/boost/1.71.0/lib/libboost_thread.so.1.71.0 ../libapicore/libapicore.a -ljsoncpp /home/jaytong/DPA_Projects/LIBS/openssl/1.1.1f/lib/libssl.so /home/jaytong/DPA_Projects/LIBS/openssl/1.1.1f/lib/libcrypto.so ../libethcore/libethcore.a ../libethash-sycl/libethash-sycl.a ../libethcore/libethcore.a ../libethash-sycl/libethash-sycl.a /home/jaytong/DPA_Projects/LIBS/ethash/0.4.3/lib/libethash.a ../libhwmon/libhwmon.a -ldl ../libdevcore/libdevcore.a /home/jaytong/DPA_Projects/LIBS/boost/1.71.0/lib/libboost_system.so.1.71.0 -lpthread libethminer-buildinfo.a /home/jaytong/DPA_Projects/LIBS/boost/1.71.0/lib/libboost_filesystem.so.1.71.0 -lstdc++ -lm -lgcc_s -lgcc -lsycl -lsycl-devicelib-host -lc -lgcc_s -lgcc /usr/lib/gcc/x86_64-linux-gnu/9/crtend.o /lib/x86_64-linux-gnu/crtn.o
 "/home/jaytong/sycl_workspace/llvm/build/install/bin/clang-offload-deps" -targets=sycl-amdgcn-amd-amdhsa-gfx900 -outputs=/tmp/main-9b930e/main-gfx900.bc /tmp/main-f470e0.out
 "/home/jaytong/sycl_workspace/llvm/build/install/bin/clang-offload-bundler" -type=a -targets=sycl-amdgcn-amd-amdhsa-gfx900 -input=../libethcore/libethcore.a -output=/tmp/libethcore-24d7b1.a -unbundle -allow-missing-bundles
 "/home/jaytong/sycl_workspace/llvm/build/install/bin/clang-offload-bundler" -type=a -targets=sycl-amdgcn-amd-amdhsa-gfx900 -input=../libpoolprotocols/libpoolprotocols.a -output=/tmp/libpoolprotocols-a0f2d9.a -unbundle -allow-missing-bundles
 "/home/jaytong/sycl_workspace/llvm/build/install/bin/clang-offload-bundler" -type=a -targets=sycl-amdgcn-amd-amdhsa-gfx900 -input=../libdevcore/libdevcore.a -output=/tmp/libdevcore-eac9ed.a -unbundle -allow-missing-bundles
 "/home/jaytong/sycl_workspace/llvm/build/install/bin/clang-offload-bundler" -type=a -targets=sycl-amdgcn-amd-amdhsa-gfx900 -input=../libapicore/libapicore.a -output=/tmp/libapicore-ed16d0.a -unbundle -allow-missing-bundles
 "/home/jaytong/sycl_workspace/llvm/build/install/bin/clang-offload-bundler" -type=a -targets=sycl-amdgcn-amd-amdhsa-gfx900 -input=../libethcore/libethcore.a -output=/tmp/libethcore-67b0ab.a -unbundle -allow-missing-bundles
 "/home/jaytong/sycl_workspace/llvm/build/install/bin/clang-offload-bundler" -type=a -targets=sycl-amdgcn-amd-amdhsa-gfx900 -input=../libethash-sycl/libethash-sycl.a -output=/tmp/libethash-sycl-ec0b3b.a -unbundle -allow-missing-bundles
 "/home/jaytong/sycl_workspace/llvm/build/install/bin/clang-offload-bundler" -type=a -targets=sycl-amdgcn-amd-amdhsa-gfx900 -input=../libethcore/libethcore.a -output=/tmp/libethcore-cdbcf9.a -unbundle -allow-missing-bundles
 "/home/jaytong/sycl_workspace/llvm/build/install/bin/clang-offload-bundler" -type=a -targets=sycl-amdgcn-amd-amdhsa-gfx900 -input=../libethash-sycl/libethash-sycl.a -output=/tmp/libethash-sycl-5d3261.a -unbundle -allow-missing-bundles
 "/home/jaytong/sycl_workspace/llvm/build/install/bin/clang-offload-bundler" -type=a -targets=sycl-amdgcn-amd-amdhsa-gfx900 -input=../libhwmon/libhwmon.a -output=/tmp/libhwmon-f4e1ab.a -unbundle -allow-missing-bundles
 "/home/jaytong/sycl_workspace/llvm/build/install/bin/clang-offload-bundler" -type=a -targets=sycl-amdgcn-amd-amdhsa-gfx900 -input=../libdevcore/libdevcore.a -output=/tmp/libdevcore-112c1c.a -unbundle -allow-missing-bundles
 "/home/jaytong/sycl_workspace/llvm/build/install/bin/llvm-link" /tmp/main-c3b415/main-gfx900.o /tmp/main-9b930e/main-gfx900.bc -o /tmp/amdgcn-link-278e37.bc --suppress-warnings
 "/home/jaytong/sycl_workspace/llvm/build/install/bin/llvm-link" --only-needed /tmp/amdgcn-link-278e37.bc /tmp/libethcore-24d7b1.a /tmp/libpoolprotocols-a0f2d9.a /tmp/libdevcore-eac9ed.a /tmp/libapicore-ed16d0.a /tmp/libethcore-67b0ab.a /tmp/libethash-sycl-ec0b3b.a /tmp/libethcore-cdbcf9.a /tmp/libethash-sycl-5d3261.a /tmp/libhwmon-f4e1ab.a /tmp/libdevcore-112c1c.a -o /tmp/main-7f87ab/main-gfx900.bc --suppress-warnings
 "/home/jaytong/sycl_workspace/llvm/build/install/bin/sycl-post-link" -split=auto -symbols -emit-exported-symbols -lower-esimd -O2 -spec-const=default -o /tmp/main-784aaa/main-gfx900.bc /tmp/main-7f87ab/main-gfx900.bc
 "/home/jaytong/sycl_workspace/llvm/build/install/bin/file-table-tform" -extract=Code -drop_titles -o /tmp/main-baf9b9/main-gfx900.bc /tmp/main-784aaa/main-gfx900.bc
 "/home/jaytong/sycl_workspace/llvm/build/install/bin/llvm-foreach" --out-ext=o --in-file-list=/tmp/main-baf9b9/main-gfx900.bc --in-replace=/tmp/main-baf9b9/main-gfx900.bc --out-file-list=/tmp/main-4ca78b/main-gfx900.o --out-replace=/tmp/main-4ca78b/main-gfx900.o -- /home/jaytong/sycl_workspace/llvm/build/install/bin/clang-15 -cc1 -triple amdgcn-amd-amdhsa -aux-triple x86_64-unknown-linux-gnu -fsycl-is-device -fdeclare-spirv-builtins -Wno-sycl-strict -sycl-std=2020 -emit-llvm-bc -disable-free -clear-ast-before-backend -main-file-name main.cpp.o -mrelocation-model pic -pic-level 1 -fhalf-no-semantic-interposition -mframe-pointer=none -ffp-contract=on -fno-rounding-math -mconstructor-aliases -aux-target-cpu x86-64 -fcuda-is-device -mllvm -amdgpu-internalize-symbols -fcuda-allow-variadic-functions -fvisibility hidden -fapply-global-visibility-to-externs -internal-isystem /home/jaytong/sycl_workspace/llvm/build/install/bin/../include/sycl -internal-isystem /home/jaytong/sycl_workspace/llvm/build/install/bin/../include -mlink-builtin-bitcode /home/jaytong/sycl_workspace/llvm/build/install/lib/clang/15.0.0/../../clc/remangled-l64-signed_char.libspirv-amdgcn--amdhsa.bc -mlink-builtin-bitcode /opt/rocm/amdgcn/bitcode/hip.bc -mlink-builtin-bitcode /opt/rocm/amdgcn/bitcode/ocml.bc -mlink-builtin-bitcode /opt/rocm/amdgcn/bitcode/ockl.bc -mlink-builtin-bitcode /opt/rocm/amdgcn/bitcode/oclc_daz_opt_off.bc -mlink-builtin-bitcode /opt/rocm/amdgcn/bitcode/oclc_unsafe_math_off.bc -mlink-builtin-bitcode /opt/rocm/amdgcn/bitcode/oclc_finite_only_off.bc -mlink-builtin-bitcode /opt/rocm/amdgcn/bitcode/oclc_correctly_rounded_sqrt_off.bc -mlink-builtin-bitcode /opt/rocm/amdgcn/bitcode/oclc_wavefrontsize64_on.bc -mlink-builtin-bitcode /opt/rocm/amdgcn/bitcode/oclc_isa_version_900.bc -target-cpu gfx900 -mllvm -treat-scalable-fixed-error-as-warning -debug-info-kind=constructor -dwarf-version=5 -debugger-tuning=gdb -v -resource-dir /home/jaytong/sycl_workspace/llvm/build/install/lib/clang/15.0.0 -O2 -Wall -Wno-unknown-pragmas -Wextra -fdebug-compilation-dir=/home/jaytong/DPA_Projects/DEV/applications.benchmarking.oneapi.onebench/ethminer/build/ethminer -ferror-limit 19 -fgnuc-version=4.2.1 -fcolor-diagnostics -vectorize-loops -vectorize-slp -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /tmp/main-4ca78b/main-gfx900.o -x ir /tmp/main-baf9b9/main-gfx900.bc
clang -cc1 version 15.0.0 based upon LLVM 15.0.0git default target x86_64-unknown-linux-gnu
 "/home/jaytong/sycl_workspace/llvm/build/install/bin/llvm-foreach" --out-ext=out --in-file-list=/tmp/main-4ca78b/main-gfx900.o --in-replace=/tmp/main-4ca78b/main-gfx900.o --out-file-list=/tmp/main-3fac75/main-gfx900.out --out-replace=/tmp/main-3fac75/main-gfx900.out -- /home/jaytong/sycl_workspace/llvm/build/install/bin/clang-offload-bundler -unbundle -type=a -input=/home/jaytong/DPA_Projects/LIBS/json/1.9.5/lib/cmake/jsoncpp/../../../lib/libjsoncpp.a -targets=sycl-amdgcn-amd-amdhsa-gfx900 -output=/tmp/libbc-jsoncpp-amdgcn-gfx900-79c07f.a -allow-missing-bundles -hip-openmp-compatible
llvm-foreach: Couldn't find replace string for output in the command.
clang-15: [0;1;31merror: [0m[1mamdgcn-link command failed with exit code 1 (use -v to see invocation)[0m

jgtong avatar Jun 27 '22 18:06 jgtong

Looks like the original command is trying to build the final executable which includes native device code for gfx900.

Initial set of clang-offload-bundler is unbundling the host code and feeding this to the GNU ld linker to generate a monolithic host binary.

Second sequence of clang-offload-bundler is extracting all device code, creating a monolithic device code LLVM IR using llvm-link and finally generate the native device code for gfx900 (last but one llvm-foreach command does this).

The last llvm-foreach command is malformed since the --in-file-list and --out-file-list aren't used in the clang-offload-bundler command.

Ideally with device native code ready, shouldn't we be calling the lld linker to create the final fat binary instead of the above llvm-foreach command?

mpanoop avatar Jul 28 '22 14:07 mpanoop

@jgtong , could you please post the output of the following command for reference here:

/home/jaytong/sycl_workspace/llvm/build/bin/clang++ -### -O3 -fsycl -fsycl-targets=amdgcn-amd-amdhsa -Xsycl-target-backend --offload-arch=gfx908 -Wall -Wno-unknown-pragmas -Wextra -stdlib=libstdc++ -fcolor-diagnostics -Qunused-arguments -O2 -g -DNDEBUG CMakeFiles/ethminer.dir/main.cpp.o -o ethminer -L/home/jaytong/DPA_Projects/LIBS/json/1.9.5/lib/cmake/jsoncpp/../../../lib -Wl,-rpath,/home/jaytong/DPA_Projects/LIBS/json/1.9.5/lib/cmake/jsoncpp/../../../lib:/home/jaytong/DPA_Projects/LIBS/boost/1.73.0/lib:/home/jaytong/DPA_Projects/LIBS/openssl/1.1.1f/lib: ../libethcore/libethcore.a ../libpoolprotocols/libpoolprotocols.a ../libdevcore/libdevcore.a libethminer-buildinfo.a /home/jaytong/DPA_Projects/LIBS/boost/1.73.0/lib/libboost_system.so.1.71.0 /home/jaytong/DPA_Projects/LIBS/boost/1.73.0/lib/libboost_thread.so.1.71.0 ../libapicore/libapicore.a -ljsoncpp /home/jaytong/DPA_Projects/LIBS/openssl/1.1.1f/lib/libssl.so /home/jaytong/DPA_Projects/LIBS/openssl/1.1.1f/lib/libcrypto.so ../libethcore/libethcore.a ../libethash-sycl/libethash-sycl.a ../libethcore/libethcore.a ../libethash-sycl/libethash-sycl.a /home/jaytong/DPA_Projects/LIBS/ethash/0.4.3/lib/libethash.a ../libhwmon/libhwmon.a -ldl ../libdevcore/libdevcore.a /home/jaytong/DPA_Projects/LIBS/boost/1.73.0/lib/libboost_system.so.1.71.0 -lpthread libethminer-buildinfo.a /home/jaytong/DPA_Projects/LIBS/boost/1.73.0/lib/libboost_filesystem.so.1.71.0

mpanoop avatar Jul 28 '22 14:07 mpanoop

@mpanoop : Here is the build log for the above command

clang version 15.0.0 (https://github.com/intel/llvm 6d04aa6462eb9b63ef2127cd56afdc408e3e32d4)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/jaytong/sycl_workspace/llvm/build/bin
"clang-offload-bundler" "-type=ao" "-targets=host-x86_64-unknown-linux-gnu" "-input=../libethcore/libethcore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocr-intel-unknown" "-input=../libethcore/libethcore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocx-intel-unknown" "-input=../libethcore/libethcore.a" "-check-section" 
"clang-offload-bundler" "-type=o" "-targets=sycl-spir64-unknown-unknown" "-input=CMakeFiles/ethminer.dir/main.cpp.o" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-spir64-unknown-unknown" "-input=../libethcore/libethcore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-spir64-unknown-unknown" "-input=../libpoolprotocols/libpoolprotocols.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-spir64-unknown-unknown" "-input=../libdevcore/libdevcore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-spir64-unknown-unknown" "-input=libethminer-buildinfo.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-spir64-unknown-unknown" "-input=../libapicore/libapicore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-spir64-unknown-unknown" "-input=../libethcore/libethcore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-spir64-unknown-unknown" "-input=../libethash-sycl/libethash-sycl.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-spir64-unknown-unknown" "-input=../libethcore/libethcore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-spir64-unknown-unknown" "-input=../libethash-sycl/libethash-sycl.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-spir64-unknown-unknown" "-input=/home/jaytong/DPA_Projects/LIBS/ethash/0.4.3/lib/libethash.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-spir64-unknown-unknown" "-input=../libhwmon/libhwmon.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-spir64-unknown-unknown" "-input=../libdevcore/libdevcore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-spir64-unknown-unknown" "-input=libethminer-buildinfo.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocx-intel-unknown" "-input=../libethcore/libethcore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocr-intel-unknown" "-input=../libethcore/libethcore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocr_emu-intel-unknown" "-input=../libethcore/libethcore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=host-x86_64-unknown-linux-gnu" "-input=../libethcore/libethcore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocx-intel-unknown" "-input=../libpoolprotocols/libpoolprotocols.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocr-intel-unknown" "-input=../libpoolprotocols/libpoolprotocols.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocr_emu-intel-unknown" "-input=../libpoolprotocols/libpoolprotocols.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=host-x86_64-unknown-linux-gnu" "-input=../libpoolprotocols/libpoolprotocols.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocx-intel-unknown" "-input=../libdevcore/libdevcore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocr-intel-unknown" "-input=../libdevcore/libdevcore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocr_emu-intel-unknown" "-input=../libdevcore/libdevcore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=host-x86_64-unknown-linux-gnu" "-input=../libdevcore/libdevcore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocx-intel-unknown" "-input=libethminer-buildinfo.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocr-intel-unknown" "-input=libethminer-buildinfo.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocr_emu-intel-unknown" "-input=libethminer-buildinfo.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=host-x86_64-unknown-linux-gnu" "-input=libethminer-buildinfo.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocx-intel-unknown" "-input=../libapicore/libapicore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocr-intel-unknown" "-input=../libapicore/libapicore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocr_emu-intel-unknown" "-input=../libapicore/libapicore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=host-x86_64-unknown-linux-gnu" "-input=../libapicore/libapicore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocx-intel-unknown" "-input=../libethcore/libethcore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocr-intel-unknown" "-input=../libethcore/libethcore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocr_emu-intel-unknown" "-input=../libethcore/libethcore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=host-x86_64-unknown-linux-gnu" "-input=../libethcore/libethcore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocx-intel-unknown" "-input=../libethash-sycl/libethash-sycl.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocr-intel-unknown" "-input=../libethash-sycl/libethash-sycl.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocr_emu-intel-unknown" "-input=../libethash-sycl/libethash-sycl.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=host-x86_64-unknown-linux-gnu" "-input=../libethash-sycl/libethash-sycl.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocx-intel-unknown" "-input=../libethcore/libethcore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocr-intel-unknown" "-input=../libethcore/libethcore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocr_emu-intel-unknown" "-input=../libethcore/libethcore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=host-x86_64-unknown-linux-gnu" "-input=../libethcore/libethcore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocx-intel-unknown" "-input=../libethash-sycl/libethash-sycl.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocr-intel-unknown" "-input=../libethash-sycl/libethash-sycl.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocr_emu-intel-unknown" "-input=../libethash-sycl/libethash-sycl.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=host-x86_64-unknown-linux-gnu" "-input=../libethash-sycl/libethash-sycl.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocx-intel-unknown" "-input=/home/jaytong/DPA_Projects/LIBS/ethash/0.4.3/lib/libethash.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocr-intel-unknown" "-input=/home/jaytong/DPA_Projects/LIBS/ethash/0.4.3/lib/libethash.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocr_emu-intel-unknown" "-input=/home/jaytong/DPA_Projects/LIBS/ethash/0.4.3/lib/libethash.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=host-x86_64-unknown-linux-gnu" "-input=/home/jaytong/DPA_Projects/LIBS/ethash/0.4.3/lib/libethash.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocx-intel-unknown" "-input=../libhwmon/libhwmon.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocr-intel-unknown" "-input=../libhwmon/libhwmon.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocr_emu-intel-unknown" "-input=../libhwmon/libhwmon.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=host-x86_64-unknown-linux-gnu" "-input=../libhwmon/libhwmon.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocx-intel-unknown" "-input=../libdevcore/libdevcore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocr-intel-unknown" "-input=../libdevcore/libdevcore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocr_emu-intel-unknown" "-input=../libdevcore/libdevcore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=host-x86_64-unknown-linux-gnu" "-input=../libdevcore/libdevcore.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocx-intel-unknown" "-input=libethminer-buildinfo.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocr-intel-unknown" "-input=libethminer-buildinfo.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=sycl-fpga_aocr_emu-intel-unknown" "-input=libethminer-buildinfo.a" "-check-section" 
"clang-offload-bundler" "-type=ao" "-targets=host-x86_64-unknown-linux-gnu" "-input=libethminer-buildinfo.a" "-check-section" 
 "/home/jaytong/sycl_workspace/llvm/build/bin/clang-offload-bundler" "-type=o" "-targets=host-x86_64-unknown-linux-gnu,sycl-amdgcn-amd-amdhsa-gfx908" "-input=CMakeFiles/ethminer.dir/main.cpp.o" "-output=/tmp/main-c1209f.o" "-output=/tmp/main-7962a8/main-gfx908.o" "-unbundle" "-allow-missing-bundles"
 "/usr/bin/ld" "-z" "relro" "--hash-style=gnu" "--eh-frame-hdr" "-m" "elf_x86_64" "-dynamic-linker" "/lib64/ld-linux-x86-64.so.2" "-o" "/tmp/main-074884.out" "/lib/x86_64-linux-gnu/crt1.o" "/lib/x86_64-linux-gnu/crti.o" "/usr/lib/gcc/x86_64-linux-gnu/9/crtbegin.o" "-L/home/jaytong/DPA_Projects/LIBS/json/1.9.5/lib/cmake/jsoncpp/../../../lib" "-L/usr/lib/gcc/x86_64-linux-gnu/9" "-L/usr/lib/gcc/x86_64-linux-gnu/9/../../../../lib64" "-L/lib/x86_64-linux-gnu" "-L/lib/../lib64" "-L/usr/lib/x86_64-linux-gnu" "-L/usr/lib/../lib64" "-L/home/jaytong/sycl_workspace/llvm/build/bin/../lib" "-L/lib" "-L/usr/lib" "/tmp/main-c1209f.o" "-rpath" "/home/jaytong/DPA_Projects/LIBS/json/1.9.5/lib/cmake/jsoncpp/../../../lib:/home/jaytong/DPA_Projects/LIBS/boost/1.73.0/lib:/home/jaytong/DPA_Projects/LIBS/openssl/1.1.1f/lib:" "../libethcore/libethcore.a" "../libpoolprotocols/libpoolprotocols.a" "../libdevcore/libdevcore.a" "libethminer-buildinfo.a" "/home/jaytong/DPA_Projects/LIBS/boost/1.73.0/lib/libboost_thread.so.1.73.0" "../libapicore/libapicore.a" "-ljsoncpp" "/home/jaytong/DPA_Projects/LIBS/openssl/1.1.1f/lib/libssl.so" "/home/jaytong/DPA_Projects/LIBS/openssl/1.1.1f/lib/libcrypto.so" "../libethcore/libethcore.a" "../libethash-sycl/libethash-sycl.a" "../libethcore/libethcore.a" "../libethash-sycl/libethash-sycl.a" "/home/jaytong/DPA_Projects/LIBS/ethash/0.4.3/lib/libethash.a" "../libhwmon/libhwmon.a" "-ldl" "../libdevcore/libdevcore.a" "/home/jaytong/DPA_Projects/LIBS/boost/1.73.0/lib/libboost_system.so.1.73.0" "-lpthread" "libethminer-buildinfo.a" "/home/jaytong/DPA_Projects/LIBS/boost/1.73.0/lib/libboost_filesystem.so.1.73.0" "-lstdc++" "-lm" "-lgcc_s" "-lgcc" "-lsycl" "-lsycl-devicelib-host" "-lc" "-lgcc_s" "-lgcc" "/usr/lib/gcc/x86_64-linux-gnu/9/crtend.o" "/lib/x86_64-linux-gnu/crtn.o"
 "/home/jaytong/sycl_workspace/llvm/build/bin/clang-offload-deps" "-targets=sycl-amdgcn-amd-amdhsa-gfx908" "-outputs=/tmp/main-bc0b67/main-gfx908.bc" "/tmp/main-074884.out"
 "/home/jaytong/sycl_workspace/llvm/build/bin/clang-offload-bundler" "-type=a" "-targets=sycl-amdgcn-amd-amdhsa-gfx908" "-input=../libethcore/libethcore.a" "-output=/tmp/libethcore-6157c2.a" "-unbundle" "-allow-missing-bundles"
 "/home/jaytong/sycl_workspace/llvm/build/bin/clang-offload-bundler" "-type=a" "-targets=sycl-amdgcn-amd-amdhsa-gfx908" "-input=../libpoolprotocols/libpoolprotocols.a" "-output=/tmp/libpoolprotocols-c93e9d.a" "-unbundle" "-allow-missing-bundles"
 "/home/jaytong/sycl_workspace/llvm/build/bin/clang-offload-bundler" "-type=a" "-targets=sycl-amdgcn-amd-amdhsa-gfx908" "-input=../libdevcore/libdevcore.a" "-output=/tmp/libdevcore-e06420.a" "-unbundle" "-allow-missing-bundles"
 "/home/jaytong/sycl_workspace/llvm/build/bin/clang-offload-bundler" "-type=a" "-targets=sycl-amdgcn-amd-amdhsa-gfx908" "-input=../libapicore/libapicore.a" "-output=/tmp/libapicore-c271bf.a" "-unbundle" "-allow-missing-bundles"
 "/home/jaytong/sycl_workspace/llvm/build/bin/clang-offload-bundler" "-type=a" "-targets=sycl-amdgcn-amd-amdhsa-gfx908" "-input=../libethcore/libethcore.a" "-output=/tmp/libethcore-56c612.a" "-unbundle" "-allow-missing-bundles"
 "/home/jaytong/sycl_workspace/llvm/build/bin/clang-offload-bundler" "-type=a" "-targets=sycl-amdgcn-amd-amdhsa-gfx908" "-input=../libethash-sycl/libethash-sycl.a" "-output=/tmp/libethash-sycl-d732f0.a" "-unbundle" "-allow-missing-bundles"
 "/home/jaytong/sycl_workspace/llvm/build/bin/clang-offload-bundler" "-type=a" "-targets=sycl-amdgcn-amd-amdhsa-gfx908" "-input=../libethcore/libethcore.a" "-output=/tmp/libethcore-5b9892.a" "-unbundle" "-allow-missing-bundles"
 "/home/jaytong/sycl_workspace/llvm/build/bin/clang-offload-bundler" "-type=a" "-targets=sycl-amdgcn-amd-amdhsa-gfx908" "-input=../libethash-sycl/libethash-sycl.a" "-output=/tmp/libethash-sycl-572cb4.a" "-unbundle" "-allow-missing-bundles"
 "/home/jaytong/sycl_workspace/llvm/build/bin/clang-offload-bundler" "-type=a" "-targets=sycl-amdgcn-amd-amdhsa-gfx908" "-input=../libhwmon/libhwmon.a" "-output=/tmp/libhwmon-c87492.a" "-unbundle" "-allow-missing-bundles"
 "/home/jaytong/sycl_workspace/llvm/build/bin/clang-offload-bundler" "-type=a" "-targets=sycl-amdgcn-amd-amdhsa-gfx908" "-input=../libdevcore/libdevcore.a" "-output=/tmp/libdevcore-3e8f4a.a" "-unbundle" "-allow-missing-bundles"
 "/home/jaytong/sycl_workspace/llvm/build/bin/llvm-link" "/tmp/main-7962a8/main-gfx908.o" "/tmp/main-bc0b67/main-gfx908.bc" "-o" "/tmp/amdgcn-link-59be25.bc" "--suppress-warnings"
 "/home/jaytong/sycl_workspace/llvm/build/bin/llvm-link" "--only-needed" "/tmp/amdgcn-link-59be25.bc" "/tmp/libethcore-6157c2.a" "/tmp/libpoolprotocols-c93e9d.a" "/tmp/libdevcore-e06420.a" "/tmp/libapicore-c271bf.a" "/tmp/libethcore-56c612.a" "/tmp/libethash-sycl-d732f0.a" "/tmp/libethcore-5b9892.a" "/tmp/libethash-sycl-572cb4.a" "/tmp/libhwmon-c87492.a" "/tmp/libdevcore-3e8f4a.a" "-o" "/tmp/main-121540/main-gfx908.bc" "--suppress-warnings"
 "/home/jaytong/sycl_workspace/llvm/build/bin/sycl-post-link" "-split=auto" "-symbols" "-emit-exported-symbols" "-lower-esimd" "-O3" "-spec-const=default" "-o" "/tmp/main-07c238/main-gfx908.bc" "/tmp/main-121540/main-gfx908.bc"
 "/home/jaytong/sycl_workspace/llvm/build/bin/file-table-tform" "-extract=Code" "-drop_titles" "-o" "/tmp/main-600d49/main-gfx908.bc" "/tmp/main-07c238/main-gfx908.bc"
 "/home/jaytong/sycl_workspace/llvm/build/bin/llvm-foreach" "--out-ext=o" "--in-file-list=/tmp/main-600d49/main-gfx908.bc" "--in-replace=/tmp/main-600d49/main-gfx908.bc" "--out-file-list=/tmp/main-f7787c/main-gfx908.o" "--out-replace=/tmp/main-f7787c/main-gfx908.o" "--" "/home/jaytong/sycl_workspace/llvm/build/bin/clang-15" "-cc1" "-triple" "amdgcn-amd-amdhsa" "-aux-triple" "x86_64-unknown-linux-gnu" "-fsycl-is-device" "-fdeclare-spirv-builtins" "-Wno-sycl-strict" "-sycl-std=2020" "-emit-llvm-bc" "-disable-free" "-clear-ast-before-backend" "-main-file-name" "main.cpp.o" "-mrelocation-model" "pic" "-pic-level" "1" "-fhalf-no-semantic-interposition" "-mframe-pointer=none" "-ffp-contract=on" "-fno-rounding-math" "-mconstructor-aliases" "-aux-target-cpu" "x86-64" "-fcuda-is-device" "-mllvm" "-amdgpu-internalize-symbols" "-fcuda-allow-variadic-functions" "-fvisibility" "hidden" "-fapply-global-visibility-to-externs" "-internal-isystem" "/home/jaytong/sycl_workspace/llvm/build/bin/../include/sycl" "-internal-isystem" "/home/jaytong/sycl_workspace/llvm/build/bin/../include" "-mlink-builtin-bitcode" "/home/jaytong/sycl_workspace/llvm/build/lib/clang/15.0.0/../../clc/remangled-l64-signed_char.libspirv-amdgcn--amdhsa.bc" "-mlink-builtin-bitcode" "/opt/rocm/amdgcn/bitcode/hip.bc" "-mlink-builtin-bitcode" "/opt/rocm/amdgcn/bitcode/ocml.bc" "-mlink-builtin-bitcode" "/opt/rocm/amdgcn/bitcode/ockl.bc" "-mlink-builtin-bitcode" "/opt/rocm/amdgcn/bitcode/oclc_daz_opt_off.bc" "-mlink-builtin-bitcode" "/opt/rocm/amdgcn/bitcode/oclc_unsafe_math_off.bc" "-mlink-builtin-bitcode" "/opt/rocm/amdgcn/bitcode/oclc_finite_only_off.bc" "-mlink-builtin-bitcode" "/opt/rocm/amdgcn/bitcode/oclc_correctly_rounded_sqrt_off.bc" "-mlink-builtin-bitcode" "/opt/rocm/amdgcn/bitcode/oclc_wavefrontsize64_on.bc" "-mlink-builtin-bitcode" "/opt/rocm/amdgcn/bitcode/oclc_isa_version_908.bc" "-mlink-builtin-bitcode" "/opt/rocm/amdgcn/bitcode/oclc_abi_version_400.bc" "-target-cpu" "gfx908" "-mllvm" "-treat-scalable-fixed-error-as-warning" "-debugger-tuning=gdb" "-resource-dir" "/home/jaytong/sycl_workspace/llvm/build/lib/clang/15.0.0" "-O3" "-Wall" "-Wno-unknown-pragmas" "-Wextra" "-fdebug-compilation-dir=/home/jaytong/DPA_Projects/DEV/applications.benchmarking.oneapi.onebench/ethminer/build_POR/ethminer" "-ferror-limit" "19" "-fgnuc-version=4.2.1" "-fcolor-diagnostics" "-vectorize-loops" "-vectorize-slp" "-faddrsig" "-o" "/tmp/main-f7787c/main-gfx908.o" "-x" "ir" "/tmp/main-600d49/main-gfx908.bc"
 "/home/jaytong/sycl_workspace/llvm/build/bin/llvm-foreach" "--out-ext=out" "--in-file-list=/tmp/main-f7787c/main-gfx908.o" "--in-replace=/tmp/main-f7787c/main-gfx908.o" "--out-file-list=/tmp/main-28e3d2/main-gfx908.out" "--out-replace=/tmp/main-28e3d2/main-gfx908.out" "--" "/home/jaytong/sycl_workspace/llvm/build/bin/clang-offload-bundler" "-unbundle" "-type=a" "-input=/home/jaytong/DPA_Projects/LIBS/json/1.9.5/lib/cmake/jsoncpp/../../../lib/libjsoncpp.a" "-targets=sycl-amdgcn-amd-amdhsa-gfx908" "-output=/tmp/libbc-jsoncpp-amdgcn-gfx908-42cf0e.a" "-allow-missing-bundles" "-hip-openmp-compatible"
 "/home/jaytong/sycl_workspace/llvm/build/bin/llvm-foreach" "--out-ext=out" "--in-file-list=/tmp/main-f7787c/main-gfx908.o" "--in-replace=/tmp/main-f7787c/main-gfx908.o" "--out-file-list=/tmp/main-28e3d2/main-gfx908.out" "--out-replace=/tmp/main-28e3d2/main-gfx908.out" "--" "/home/jaytong/sycl_workspace/llvm/build/bin/lld" "-flavor" "gnu" "--no-undefined" "-shared" "-plugin-opt=-amdgpu-internalize-symbols" "-plugin-opt=mcpu=gfx908" "-plugin-opt=O3" "-o" "/tmp/main-28e3d2/main-gfx908.out" "/tmp/main-f7787c/main-gfx908.o" "/tmp/libbc-jsoncpp-amdgcn-gfx908-42cf0e.a"
 "/home/jaytong/sycl_workspace/llvm/build/bin/llvm-foreach" "--out-ext=hipfb" "--in-file-list=/tmp/main-28e3d2/main-gfx908.out" "--in-replace=/tmp/main-28e3d2/main-gfx908.out" "--out-file-list=/tmp/main-405129/main-gfx908.hipfb" "--out-replace=/tmp/main-405129/main-gfx908.hipfb" "--" "/home/jaytong/sycl_workspace/llvm/build/bin/clang-offload-bundler" "-type=o" "-bundle-align=4096" "-targets=host-x86_64-unknown-linux,hipv4-amdgcn-amd-amdhsa--gfx908" "-input=/dev/null" "-input=/tmp/main-28e3d2/main-gfx908.out" "-output=/tmp/main-405129/main-gfx908.hipfb"
 "/home/jaytong/sycl_workspace/llvm/build/bin/file-table-tform" "-replace=Code,Code" "-o" "/tmp/main-99996d/main-gfx908.table" "/tmp/main-07c238/main-gfx908.bc" "/tmp/main-405129/main-gfx908.hipfb"
 "/home/jaytong/sycl_workspace/llvm/build/bin/clang-offload-wrapper" "-o=/tmp/wrapper-504331.bc" "-host=x86_64-unknown-linux-gnu" "-compile-opts=--offload-arch=gfx908" "-target=amdgcn" "-kind=sycl" "-batch" "/tmp/main-99996d/main-gfx908.table"
 "/home/jaytong/sycl_workspace/llvm/build/bin/llc" "-filetype=obj" "-o" "/tmp/ethminer-wrapper-2649eb/ethminer-wrapper-gfx908.o" "/tmp/wrapper-504331.bc" "-relocation-model=pic"
 "/usr/bin/ld" "-z" "relro" "--hash-style=gnu" "--eh-frame-hdr" "-m" "elf_x86_64" "-dynamic-linker" "/lib64/ld-linux-x86-64.so.2" "-o" "ethminer" "/lib/x86_64-linux-gnu/crt1.o" "/lib/x86_64-linux-gnu/crti.o" "/usr/lib/gcc/x86_64-linux-gnu/9/crtbegin.o" "-L/home/jaytong/DPA_Projects/LIBS/json/1.9.5/lib/cmake/jsoncpp/../../../lib" "-L/usr/lib/gcc/x86_64-linux-gnu/9" "-L/usr/lib/gcc/x86_64-linux-gnu/9/../../../../lib64" "-L/lib/x86_64-linux-gnu" "-L/lib/../lib64" "-L/usr/lib/x86_64-linux-gnu" "-L/usr/lib/../lib64" "-L/home/jaytong/sycl_workspace/llvm/build/bin/../lib" "-L/lib" "-L/usr/lib" "/tmp/main-c1209f.o" "-rpath" "/home/jaytong/DPA_Projects/LIBS/json/1.9.5/lib/cmake/jsoncpp/../../../lib:/home/jaytong/DPA_Projects/LIBS/boost/1.73.0/lib:/home/jaytong/DPA_Projects/LIBS/openssl/1.1.1f/lib:" "../libethcore/libethcore.a" "../libpoolprotocols/libpoolprotocols.a" "../libdevcore/libdevcore.a" "libethminer-buildinfo.a" "/home/jaytong/DPA_Projects/LIBS/boost/1.73.0/lib/libboost_thread.so.1.73.0" "../libapicore/libapicore.a" "-ljsoncpp" "/home/jaytong/DPA_Projects/LIBS/openssl/1.1.1f/lib/libssl.so" "/home/jaytong/DPA_Projects/LIBS/openssl/1.1.1f/lib/libcrypto.so" "../libethcore/libethcore.a" "../libethash-sycl/libethash-sycl.a" "../libethcore/libethcore.a" "../libethash-sycl/libethash-sycl.a" "/home/jaytong/DPA_Projects/LIBS/ethash/0.4.3/lib/libethash.a" "../libhwmon/libhwmon.a" "-ldl" "../libdevcore/libdevcore.a" "/home/jaytong/DPA_Projects/LIBS/boost/1.73.0/lib/libboost_system.so.1.73.0" "-lpthread" "libethminer-buildinfo.a" "/home/jaytong/DPA_Projects/LIBS/boost/1.73.0/lib/libboost_filesystem.so.1.73.0" "/tmp/ethminer-wrapper-2649eb/ethminer-wrapper-gfx908.o" "-lstdc++" "-lm" "-lgcc_s" "-lgcc" "-lsycl" "-lsycl-devicelib-host" "-lc" "-lgcc_s" "-lgcc" "/usr/lib/gcc/x86_64-linux-gnu/9/crtend.o" "/lib/x86_64-linux-gnu/crtn.o"

jgtong avatar Jul 28 '22 15:07 jgtong

Thanks @jgtong. Looks like the clang-offload-bundler in the malformed llvm-foreach command (which is failing) should be invoked outside of llvm-foreach command considering the sequence of commands which follows after this malformed command. We need to check the sequence of commands which compiler driver is creating.

mpanoop avatar Jul 28 '22 18:07 mpanoop

@jgtong, this should be addressed with https://github.com/intel/llvm/pull/6498, can you confirm and close as appropriate?

mdtoguchi avatar Aug 12 '22 16:08 mdtoguchi

@jgtong, this should be addressed with #6498, can you confirm and close as appropriate?

@mdtoguchi : I can confirm that this issue has been resolved. Thank you!

jgtong avatar Aug 12 '22 16:08 jgtong