LLNL
use oneMKL to provide CPU LAPACK functions
Originally posted by @gardner48 in https://github.com/LLNL/sundials/issues/173#issuecomment-1176270242
Naively following the first suggestion in #173 by doing something like cmake ... -DBLA_VENDOR=Intel10_64ilp ... results in
CMake Warning:
Manually-specified variables were not used by the project:
BLA_VENDOR
FindBLAS exists on my system (ubuntu 22.04) but apparently is not implemented/used by sundials. FWIW FindBLAS does work with MKL for at least one other lib (suitesparse) on my system.
The second suggestion, specify the lapack libraries along with appropriate (global) cflags, works; however results in all built dynamic libraries having links to MKL. There does not seem to be cmake options for linking and c flags specific to lapack. One can do two builds e.g. one for all non lapack libraries, a second for only the two sundials lapack libraries, and then use the two lib sets together (this is what I've done and so far works) but seems like a kludge.
Can the sundials cmake build system better handle linking to MKL? Have I missed some cmake option or should I label this issuse as a feature request?
Did you set ENABLE_LAPACK to ON? BLA_VENDOR should be used. Since SUNDIALS doesn't ever use BLAS without also needing LAPACK, we only call FindLAPACK.
Thank you for responding. I tried multiple ways, most using the cmake command line and nearly all with -DENABLE_LAPACK=ON. That said, I also tried the ccmake curses gui with a simpler config than I give below.
For the cmake cli, I use a script to generate the cmake command, they are long but here are examples:
cmake cli examples
With -DBLA_VENDOR=Intel10_64ilp
/usr/bin/cmake -DBUILD_STATIC_LIBS=OFF -DENABLE_OPENMP=ON -DCMAKE_INSTALL_PREFIX:PATH=/home/ul/.local -DCMAKE_INSTALL_LIBDIR:PATH=lib/sundials-6.6.1.mkl -DCMAKE_INSTALL_INCLUDEDIR:PATH=include/sundials-6.6.1.mkl -DEXAMPLES_INSTALL_PATH:PATH=/home/ul/.local/examples/sundials-6.6.1.mkl -DENABLE_KLU=ON -DKLU_INCLUDE_DIR=/home/ul/.local/include/suitesparse-7.2.0.mkl -DKLU_LIBRARY_DIR=/home/ul/.local/lib/suitesparse-7.2.0.mkl -DENABLE_LAPACK=ON -DBLA_VENDOR=Intel10_64ilp -DCMAKE_C_FLAGS="-march=native -fPIC -O3 -DNDEBUG" ../
Specifying the lapack libraries (works but is a pain to make an install that does not have all libraries linked to mkl):
/usr/bin/cmake -DBUILD_STATIC_LIBS=OFF -DENABLE_OPENMP=ON -DCMAKE_INSTALL_PREFIX:PATH=/home/ul/.local -DCMAKE_INSTALL_LIBDIR:PATH=lib/sundials-6.6.1.mkl -DCMAKE_INSTALL_INCLUDEDIR:PATH=include/sundials-6.6.1.mkl -DEXAMPLES_INSTALL_PATH:PATH=/home/ul/.local/examples/sundials-6.6.1.mkl -DENABLE_KLU=ON -DKLU_INCLUDE_DIR=/home/ul/.local/include/suitesparse-7.2.0.mkl -DKLU_LIBRARY_DIR=/home/ul/.local/lib/suitesparse-7.2.0.mkl -DENABLE_LAPACK=ON -DLAPACK_LIBRARIES="/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_core.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_intel_ilp64.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_gf_ilp64.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_gnu_thread.so" -DCMAKE_C_FLAGS="-march=native -fPIC -O3 -DNDEBUG -DMKL_ILP64 -m64 -I/opt/intel/oneapi/mkl/2023.2.0/include -L/opt/intel/oneapi/mkl/2023.2.0/lib/intel64 -Wl,--no-as-needed -lmkl_intel_ilp64 -lmkl_gnu_thread -lmkl_core -lgomp -lpthread -lm -ldl" ../
My appologies, I just reran the cmake cli example with -DBLA_VENDOR=Intel10_64ilp now and noticed the error output:
... <truncated>
-- Found Threads: TRUE
-- Looking for Fortran sgemm
-- Looking for Fortran sgemm - found
-- Found BLAS: /opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_gf_ilp64.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_gnu_thread.so;/opt/intel/oneapi/mkl/2023.2.0/lib/intel64/libmkl_core.so;/usr/lib/gcc/x86_64-linux-gnu/11/libgomp.so;-lm;-ldl
CMake Error at /usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
Could NOT find LAPACK (missing: LAPACK_LIBRARIES)
Call Stack (most recent call first):
/usr/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
/usr/share/cmake-3.22/Modules/FindLAPACK.cmake:744 (find_package_handle_standard_args)
cmake/tpl/SundialsLapack.cmake:50 (find_package)
cmake/SundialsSetupTPLs.cmake:122 (include)
CMakeLists.txt:171 (include)
specifying the lapack libraries (without the mkl link advisor flags included in -DCMAKE_C_FLAGS) gives me
...
ERROR: SUNDIALS interface to LAPACK is not functional.
...
I can always add the link advisor flags, but then there is no point to using -DBLA_VENDOR=Intel10_64ilp is there?
Let me know if you have a suggestion for me to try.
For the record, I'm using sundials 6.6.1 and oneMKL 2023.0 Update 2 Product build 20230613 installed via sudo apt install intel-oneapi-mkl intel-oneapi-mkl-devel