deepmodeling
Error while using AMD AOCC/AOCL to build ABACUS
Details
While using AMD AOCC/AOCL (5.0.0.1 version, accessed today) to build ABACUS, I encounter error in the last linking step
[100%] Built target lr
[100%] Building CXX object CMakeFiles/abacus.dir/source/main.cpp.o
[100%] Linking CXX executable abacus
ld.lld: error: undefined symbol: container::kernels::set_memory<std::complex<double>, container::DEVICE_GPU>::operator()(std::complex<double>*, std::complex<double> const&, unsigned long const&)>>> referenced by propagator.h:94 (/home/mikoto/tests/test2/abacus-develop/source/module_hamilt_lcao/module_tddft/propagator.h:94)
>>> source/module_hamilt_lcao/module_tddft/CMakeFiles/tddft.dir/propagator_cn2.cpp.o:(container::Tensor module_tddft::create_identity_matrix<std::complex<double>>(int, container::DeviceType))
>>> did you mean: container::kernels::set_memory<std::complex<double>, container::DEVICE_CPU>::operator()(std::complex<double>*, std::complex<double> const&, unsigned long const&)
>>> defined in: source/module_base/module_container/libcontainer.a(memory_impl.cpp.o)
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[2]: *** [CMakeFiles/abacus.dir/build.make:1198: abacus] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:1247: CMakeFiles/abacus.dir/all] Error 2
gmake: *** [Makefile:136: all] Error 2
So:
- How can we try to solve this problem?
- Does ABACUS support compilation like
-
- gcc-MKL
-
- AOCC-AOCL
-
- gcc-AOCL
Other compilation setting: gcc/g++/gfortran 11.4 clang/clang++:
AMD clang version 17.0.6 (CLANG: AOCC_5.0.0-Build#1377 2024_09_24)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /data/softwares/AMD/5.0.0-aocl-aocc/aocc/bin
LAPACK: OpenBLAS (by toolchain) or AOCL ScaLAPACK, FFTW：From AOCL ELPA: compiled by toolchain
Hardware: AMD EPYC 7b12 OS: Ubuntu 22.04
Notice: flang cannot be used, which will lead to error in build ELPA.
flang: error: unsupported option '--whole-archive'
flang: error: unsupported option '--no-whole-archive'
flang: error: unknown argument: '-soname'
flang: error: no such file or directory: 'libelpa_openmp.so.19'
Have you read FAQ on the online manual http://abacus.deepmodeling.com/en/latest/community/faq.html
- [x] Yes, I have read the FAQ part on online manual.
Task list for Issue attackers (only for developers)
- [ ] Understand the problem or question described by the user.
- [ ] Check if the issue is a known problem or has been addressed in the documentation.
- [ ] Test the issue or problem on a similar system or environment, if possible.
- [ ] Identify the root cause or provide clarification on the user's question.
- [ ] Provide a step-by-step guide, including any necessary resources, to resolve the issue or answer the question.
- [ ] If the issue is related to documentation, update the documentation to prevent future confusion (optional).
- [ ] If the issue is related to code, consider implementing a fix or improvement (optional).
- [ ] Review and incorporate any relevant feedback from users or developers.
- [ ] Ensure the user's issue is resolved or their question is answered and close the ticket.
GCC-AOCL installation of ABACUS can be done via toolchain, but I need to do lapack link in toolchain for automatic installation
GCC-AOCL installation of ABACUS can be done via toolchain, but I need to do lapack link in toolchain for automatic installation
GCC-AOCL version of ABACUS can be compiled while OpenMPI should be compiled by AOCC-AOCL and keep ELPA be compiled by GCC-OpenBLAS. The SCF effciency of this GCC-AOCL version is slighly lower than GCC-OpenBLAS version of abacus. The buliding script is like:
#!/bin/bash
#SBATCH -J build
#SBATCH -N 1
#SBATCH -n 16
#SBATCH -o install.log
#SBATCH -e install.err
# install ABACUS with libxc and deepks
# JamesMisaka in 2023.08.31
# Build ABACUS by gnu-AOCL
module load openmpi/5.0.6-amd5.0.0 cereal cmake
TOOL=$(pwd)
VER=250302
ABACUS_DIR=../../abacus-$VER
#source ./install/setup
cd $ABACUS_DIR
#PREFIX=.
PREFIX=/data/softwares/abacus/$VER-aocl
BUILD_DIR=build_abacus_aocl
APPDIR=/data/softwares
LIBDIR=/data/libraries
LAPACK=/data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib
SCALAPACK=/data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib
ELPA=$LIBDIR/elpa/2025.01.001-gcc/cpu
FFTW3=/data/softwares/AMD/5.0.0-aocl-aocc/aocl
CEREAL=$LIBDIR/cereal/1.3.2
LIBXC=$LIBDIR/libxc/7.0.0-gcc
LIBNPY=$LIBDIR/libnpy/1.0.1/include
LIBRI=$LIBDIR/libri/0.2.0
LIBCOMM=$LIBDIR/libcomm/0.1.0
cmake -B $BUILD_DIR -DCMAKE_INSTALL_PREFIX=$PREFIX \
-DCMAKE_CXX_COMPILER=g++ \
-DMPI_CXX_COMPILER=mpicxx \
-DLAPACK_DIR=$LAPACK \
-DSCALAPACK_DIR=$SCALAPACK \
-DELPA_DIR=$ELPA \
-DFFTW3_DIR=$FFTW3 \
-DCEREAL_INCLUDE_DIR=$CEREAL \
-DLibxc_DIR=$LIBXC \
-DENABLE_LCAO=ON \
-DENABLE_LIBXC=ON \
-DUSE_OPENMP=ON \
-DENABLE_ASAN=OFF \
-DUSE_ELPA=ON \
-DENABLE_LIBRI=OFF \
cmake --build $BUILD_DIR -j `nproc` | tee build.log
cmake --install $BUILD_DIR | tee install.log
but if change the -DCMAKE_CXX_COMPILER to clang++ in AOCC, error occur like above.
It seems that while using clang++, some package related to GPU-ABACUS need to be deal with in some way. However, the hardware in this machine is only AMD-EPYC-7b12 without any GPU
@caic99 @dzzz2001 kindly ask whether you have some idea ?
error in the last linking step
This largely blames to the difference of grammars which is not a part of ISO C++ between clang and gcc. Generally, it only requires little modification on codes.
Notice: flang cannot be used, which will lead to error in build ELPA.
It would be better to submit a issue in the repo of ELPA if they did not mention the availability on AMD compilers.
The SCF effciency of this GCC-AOCL version is slighly lower
With LCAO or PW basis?
The SCF effciency of this GCC-AOCL version is slighly lower
With LCAO or PW basis?
LCAO basis in Fe80C36 surface system, run by MPI16-OMP1 in AMD-epyc-7b12
- gcc-openblas: 40s/scf-step
- gcc-aocl (elpa from gcc-openblas): 41s/scf-step
- Intel-OneAPI: 45s/scf-step (after libfakeintel.so crack)
It would be better to submit a issue in the repo of ELPA if they did not mention the availability on AMD compilers.
Or just try ignoring those args if you believe it's OK to do so. Plz see https://github.com/clangd/clangd/issues/662
LCAO basis
I would suggest you checking the underlying math libs used by elpa to see if functions from AOCL is really in use. Plus, it is a common practice to compile with a standard BLAS implementation and use a vendor library in linking stage / at runtime.
GCC-AOCL installation of ABACUS can be done via toolchain, but I need to do lapack link in toolchain for automatic installation
Which is done and will be upload in the existing toolchain PR
LCAO basis
I would suggest you checking the underlying math libs used by elpa to see if functions from AOCL is really in use. Plus, it is a common practice to compile with a standard BLAS implementation and use a vendor library in linking stage / at runtime.
@caic99 Thanks for your suggestion, after fully support AOCL in toolchain, I've complied another abacus and find that this gcc-AOCL version of ABACUS have better efficiency in LCAO-genelpa SCF calculation. Here is a ranking list after check-out, showing the time comsumed in each scf step (not include the first and the last scf step in each ion step)
Task: MPI16-OMP1 LCAO-genelpa for Fe5C2(510) [Fe80C36], ABACUS commit 1fa5e3a8c , Hardware: AMD-EPYC-7b12
- gcc-aocl: 38s/scf-step
- gcc-openblas 40s/scf-step
- aocc-openblas-aocl （38-42）s/scf-step （unstable）
- intel-oneapi 46s/scf-step
And here is the link libs of ABACUS in gcc-aocl
linux-vdso.so.1 (0x00007ffd813fe000)
/data/libraries/fakeintel/libfakeintel.so (0x000079512e365000)
libelpa_openmp.so.19 => /data/libraries/elpa/2025.01.001-amd/cpu/lib/libelpa_openmp.so.19 (0x000079512d400000)
libfftw3.so.3 => /data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib/libfftw3.so.3 (0x000079512cc00000)
libscalapack.so => /data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib/libscalapack.so (0x000079512c400000)
libfftw3_omp.so.3 => /data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib/libfftw3_omp.so.3 (0x000079512e358000)
libxc.so.15 => /data/libraries/libxc/7.0.0-gcc/lib/libxc.so.15 (0x000079512b600000)
libmpi.so.40 => /data/softwares/openmpi/5.0.6-amd5/lib/libmpi.so.40 (0x000079512b200000)
libomp.so => /data/softwares/AMD/5.0.0-aocl-aocc/aocc/lib/libomp.so (0x000079512ac00000)
libblis.so.5 => /data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib/libblis.so.5 (0x000079512a000000)
libflame.so => /data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib/libflame.so (0x0000795129000000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x0000795128c00000)
libm.so.6 => /usr/lib/x86_64-linux-gnu/libm.so.6 (0x000079512e26f000)
libgcc_s.so.1 => /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 (0x000079512e24f000)
libc.so.6 => /usr/lib/x86_64-linux-gnu/libc.so.6 (0x0000795128800000)
libmpi_mpifh.so.40 => /data/softwares/openmpi/5.0.6-amd5/lib/libmpi_mpifh.so.40 (0x000079512d78a000)
libgfortran.so.5 => /usr/lib/x86_64-linux-gnu/libgfortran.so.5 (0x0000795128400000)
libaoclutils.so => /data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib/libaoclutils.so (0x000079512d762000)
libblis-mt.so.5 => /data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib/libblis-mt.so.5 (0x0000795127800000)
libmpi_usempif08.so.40 => /data/softwares/openmpi/5.0.6-amd5/lib/libmpi_usempif08.so.40 (0x000079512d3c7000)
libmpi_usempi_ignore_tkr.so.40 => /data/softwares/openmpi/5.0.6-amd5/lib/libmpi_usempi_ignore_tkr.so.40 (0x000079512d755000)
libflang.so => /data/softwares/AMD/5.0.0-aocl-aocc/aocc/lib/libflang.so (0x0000795127200000)
libflangrti.so => /data/softwares/AMD/5.0.0-aocl-aocc/aocc/lib/libflangrti.so (0x000079512d748000)
libpgmath.so => /data/softwares/AMD/5.0.0-aocl-aocc/aocc/lib/libpgmath.so (0x0000795126e00000)
libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x000079512cbb8000)
librt.so.1 => /usr/lib/x86_64-linux-gnu/librt.so.1 (0x000079512e246000)
libpthread.so.0 => /usr/lib/x86_64-linux-gnu/libpthread.so.0 (0x000079512e241000)
libopen-pal.so.80 => /data/softwares/openmpi/5.0.6-amd5/lib/libopen-pal.so.80 (0x000079512caf7000)
libpmix.so.2 => /data/softwares/openmpi/5.0.6-amd5/lib/libpmix.so.2 (0x0000795126a00000)
libmunge.so.2 => /usr/lib/x86_64-linux-gnu/libmunge.so.2 (0x000079512d73d000)
libevent_core-2.1.so.7 => /data/softwares/openmpi/5.0.6-amd5/lib/libevent_core-2.1.so.7 (0x000079512d392000)
libevent_pthreads-2.1.so.7 => /data/softwares/openmpi/5.0.6-amd5/lib/libevent_pthreads-2.1.so.7 (0x000079512d738000)
libhwloc.so.15 => /usr/lib/x86_64-linux-gnu/libhwloc.so.15 (0x000079512b5a4000)
/lib64/ld-linux-x86-64.so.2 (0x000079512e36c000)
libdl.so.2 => /usr/lib/x86_64-linux-gnu/libdl.so.2 (0x000079512caf2000)
libudev.so.1 => /usr/lib/x86_64-linux-gnu/libudev.so.1 (0x000079512cac8000)
Looking forward for this issue being solved and running AOCC-AOCL version of ABACUS in AMD hardware
It would be better to submit a issue in the repo of ELPA if they did not mention the availability on AMD compilers.
Or just try ignoring those args if you believe it's OK to do so. Plz see clangd/clangd#662
I've done this with add a command in toolchain when in build dir of ELPA
sed -i ./libtool \
-e 's/\\$wl-soname //g' \
-e 's/\\$wl--whole-archive\\$convenience \\$wl--no-whole-archive//g' \
-e 's/\\$wl\\$soname //g'
Then the error is supressed lol
It would be better to submit a issue in the repo of ELPA if they did not mention the availability on AMD compilers.
Or just try ignoring those args if you believe it's OK to do so. Plz see clangd/clangd#662
I've done this with add a command in toolchain when in build dir of ELPA
sed -i ./libtool \ -e 's/\\$wl-soname //g' \ -e 's/\\$wl--whole-archive\\$convenience \\$wl--no-whole-archive//g' \ -e 's/\\$wl\\$soname //g'Then the error is supressed lol
ABACUS GCC-AOCL after this operation can be compiled, but have much lower efficiency in 52s/scf-step. It seems that there are some problems in flang-compiled ELPA.
ldd information
(base) mikoto@amd-cpu:/data/softwares/abacus/250302-aocl/bin$ ldd abacus
linux-vdso.so.1 (0x00007fff74360000)
/data/libraries/fakeintel/libfakeintel.so (0x00007fdb8df15000)
/data/libraries/elpa/2025.01.001-amd/cpu/lib/libelpa_openmp.so (0x00007fdb8ce00000)
libfftw3.so.3 => /data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib/libfftw3.so.3 (0x00007fdb8c600000)
libscalapack.so => /data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib/libscalapack.so (0x00007fdb8be00000)
libfftw3_omp.so.3 => /data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib/libfftw3_omp.so.3 (0x00007fdb8df08000)
libxc.so.15 => /data/libraries/libxc/7.0.0-gcc/lib/libxc.so.15 (0x00007fdb8ac00000)
libmpi.so.40 => /data/softwares/openmpi/5.0.6-amd5.0.0/lib/libmpi.so.40 (0x00007fdb8a800000)
libomp.so => /data/softwares/AMD/5.0.0-aocl-aocc/aocc/lib/libomp.so (0x00007fdb8a200000)
libblis.so.5 => /data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib/libblis.so.5 (0x00007fdb89600000)
libflame.so => /data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib/libflame.so (0x00007fdb88600000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fdb88200000)
libm.so.6 => /usr/lib/x86_64-linux-gnu/libm.so.6 (0x00007fdb8d319000)
libgcc_s.so.1 => /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fdb8dee6000)
libc.so.6 => /usr/lib/x86_64-linux-gnu/libc.so.6 (0x00007fdb87e00000)
libmpi_usempif08.so.40 => /data/softwares/openmpi/5.0.6-amd5.0/lib/libmpi_usempif08.so.40 (0x00007fdb8deb3000)
libmpi_usempi_ignore_tkr.so.40 => /data/softwares/openmpi/5.0.6-amd5.0/lib/libmpi_usempi_ignore_tkr.so.40 (0x00007fdb8dea8000)
libmpi_mpifh.so.40 => /data/softwares/openmpi/5.0.6-amd5.0/lib/libmpi_mpifh.so.40 (0x00007fdb8d2a8000)
libopen-pal.so.80 => /data/softwares/openmpi/5.0.6-amd5.0/lib/libopen-pal.so.80 (0x00007fdb8c53f000)
libpmix.so.2 => /usr/lib/x86_64-linux-gnu/libpmix.so.2 (0x00007fdb8bc43000)
libmunge.so.2 => /usr/lib/x86_64-linux-gnu/libmunge.so.2 (0x00007fdb8de9f000)
libevent_core-2.1.so.7 => /usr/lib/x86_64-linux-gnu/libevent_core-2.1.so.7 (0x00007fdb8de6a000)
libevent_pthreads-2.1.so.7 => /usr/lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.7 (0x00007fdb8de63000)
libhwloc.so.15 => /usr/lib/x86_64-linux-gnu/libhwloc.so.15 (0x00007fdb8d24c000)
libflang.so => /data/softwares/AMD/5.0.0-aocl-aocc/aocc/lib/libflang.so (0x00007fdb87800000)
libflangrti.so => /data/softwares/AMD/5.0.0-aocl-aocc/aocc/lib/libflangrti.so (0x00007fdb8de56000)
libpgmath.so => /data/softwares/AMD/5.0.0-aocl-aocc/aocc/lib/libpgmath.so (0x00007fdb87400000)
libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007fdb8cdb8000)
libaoclutils.so => /data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib/libaoclutils.so (0x00007fdb8d224000)
libblis-mt.so.5 => /data/softwares/AMD/5.0.0-aocl-aocc/aocl/lib/libblis-mt.so.5 (0x00007fdb86800000)
librt.so.1 => /usr/lib/x86_64-linux-gnu/librt.so.1 (0x00007fdb8de4f000)
libpthread.so.0 => /usr/lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fdb8de4a000)
/lib64/ld-linux-x86-64.so.2 (0x00007fdb8df1c000)
libdl.so.2 => /usr/lib/x86_64-linux-gnu/libdl.so.2 (0x00007fdb8de45000)
libcurl-gnutls.so.4 => /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 (0x00007fdb8ab5e000) libudev.so.1 => /usr/lib/x86_64-linux-gnu/libudev.so.1 (0x00007fdb8c515000)
libnghttp2.so.14 => /usr/lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007fdb8c4eb000)
libidn2.so.0 => /usr/lib/x86_64-linux-gnu/libidn2.so.0 (0x00007fdb8cd97000)
librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007fdb8c4cc000)
libssh.so.4 => /usr/lib/x86_64-linux-gnu/libssh.so.4 (0x00007fdb8a793000)
libpsl.so.5 => /usr/lib/x86_64-linux-gnu/libpsl.so.5 (0x00007fdb8c4b8000)
libnettle.so.8 => /usr/lib/x86_64-linux-gnu/libnettle.so.8 (0x00007fdb8ab18000)
libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007fdb8a5a8000)
libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007fdb8a554000) libldap-2.5.so.0 => /usr/lib/x86_64-linux-gnu/libldap-2.5.so.0 (0x00007fdb8a1a1000)
liblber-2.5.so.0 => /usr/lib/x86_64-linux-gnu/liblber-2.5.so.0 (0x00007fdb8bc32000)
libzstd.so.1 => /usr/lib/x86_64-linux-gnu/libzstd.so.1 (0x00007fdb8a0d2000)
libbrotlidec.so.1 => /usr/lib/x86_64-linux-gnu/libbrotlidec.so.1 (0x00007fdb8bc24000)
libz.so.1 => /usr/lib/x86_64-linux-gnu/libz.so.1 (0x00007fdb8a538000)
libunistring.so.2 => /usr/lib/x86_64-linux-gnu/libunistring.so.2 (0x00007fdb88456000)
libhogweed.so.6 => /usr/lib/x86_64-linux-gnu/libhogweed.so.6 (0x00007fdb8a08a000)
libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007fdb8817e000)
libcrypto.so.3 => /usr/lib/x86_64-linux-gnu/libcrypto.so.3 (0x00007fdb86200000)
libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007fdb88043000)
libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007fdb8a520000)
libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007fdb87d35000)
libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007fdb8a05b000)
libcom_err.so.2 => /usr/lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007fdb8cd91000)
libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007fdb8a512000) libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007fdb8a040000)
libbrotlicommon.so.1 => /usr/lib/x86_64-linux-gnu/libbrotlicommon.so.1 (0x00007fdb88433000)
libffi.so.8 => /usr/lib/x86_64-linux-gnu/libffi.so.8 (0x00007fdb8a505000)
libkeyutils.so.1 => /usr/lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007fdb8a039000)
libresolv.so.2 => /usr/lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fdb895ec000)