Perlmutter: Could NOT find TPLCUBLAS (missing: CUDA_CUBLAS_LIBRARIES)

[jczhang07]

On Perlmutter at NERSC, I could not build KK with TPL, even with correct library paths like

-DKokkosKernels_CUBLAS_ROOT=/opt/nvidia/hpc_sdk/Linux_x86_64/21.11/math_libs/11.5 
-DKokkosKernels_CUSPARSE_ROOT=/opt/nvidia/hpc_sdk/Linux_x86_64/21.11/math_libs/11.5

I had cudatoolkit/11.5, cmake/3.22.0. The error message was

-- Found CUDA: /opt/nvidia/hpc_sdk/Linux_x86_64/21.11/cuda/11.5 (found version "11.5")
-- Configuring incomplete, errors occurred!
See also "/global/u1/j/jczhang/petsc/arch-kokkos-dbg/externalpackages/git.kokkos-kernels/petsc-build/CMakeFiles/CMakeOutput.log".CMake Warning at /global/homes/j/jczhang/petsc/arch-kokkos-dbg/lib/cmake/Kokkos/KokkosConfigCommon.cmake:34 (MESSAGE):
  The installed Kokkos configuration does not support CXX extensions.
  Forcing -DCMAKE_CXX_EXTENSIONS=Off
Call Stack (most recent call first):
  /global/homes/j/jczhang/petsc/arch-kokkos-dbg/lib/cmake/Kokkos/KokkosConfig.cmake:56 (INCLUDE)
  CMakeLists.txt:109 (FIND_PACKAGE)


  CMake Error at /global/common/software/nersc/pm-2021q4/sw/cmake-3.22.0/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
    Could NOT find TPLCUBLAS (missing: CUDA_CUBLAS_LIBRARIES)
Call Stack (most recent call first):
  /global/common/software/nersc/pm-2021q4/sw/cmake-3.22.0/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  cmake/Modules/FindTPLCUBLAS.cmake:13 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  cmake/kokkoskernels_tpls.cmake:376 (FIND_PACKAGE)
  cmake/kokkoskernels_tpls.cmake:495 (KOKKOSKERNELS_IMPORT_TPL)
  CMakeLists.txt:196 (INCLUDE)

[sblondel]

I have exactly the same error on Swing at ANL with 1) nvhpc/21.5-oxhtyof 2) cmake/3.21.1-e5i6eks .

[lucbv]

I'll go check what happens on Perlmutter, thanks for reporting this

[cwpearson]

Hi,

I'm able to build the current Kokkos Kernels develop on perlmutter in the following configuration:

export CUDAARCHS="80" # for cmake 3.20+
export MPICH_GPU_SUPPORT_ENABLED=1
export CRAY_ACCEL_TARGET=nvidia80

export NVCC_WRAPPER_DEFAULT_COMPILER=CC

module load PrgEnv-gnu
module load cmake/3.22.0
module load cudatoolkit
module load cpe-cuda

yielding

Currently Loaded Modules:
  1) craype-x86-milan                      10) cudatoolkit/11.5
  2) libfabric/1.11.0.4.114                11) PrgEnv-gnu/8.3.3
  3) craype-network-ofi                    12) cray-dsmml/0.2.2
  4) xpmem/2.3.2-2.2_7.5__g93dd7ee.shasta  13) cray-libsci/21.08.1.2
  5) xalt/2.10.2                           14) cray-mpich/8.1.15
  6) darshan/3.3.1                         15) craype/2.7.15
  7) cmake/3.22.0                          16) gcc/10.3.0
  8) Nsight-Compute/2022.1.1               17) perftools-base/22.04.0
  9) Nsight-Systems/2022.2.1               18) cpe-cuda/22.04

NVCC_WRAPPER should be the path to kokkos/bin/nvcc_wrapper

cmake .. \
-DCMAKE_CXX_COMPILER=${NVCC_WRAPPER} \
-DCMAKE_BUILD_TYPE=Release \
-DKokkos_ENABLE_HWLOC=Off \
-DKokkosKernels_INST_COMPLEX_FLOAT=OFF \
-DKokkosKernels_INST_DOUBLE=ON \
-DKokkosKernels_INST_FLOAT=ON \
-DKokkosKernels_INST_OFFSET_INT=OFF \
-DKokkosKernels_ENABLE_TESTS=ON \
-DKokkos_ARCH_AMPERE80=ON \
-DKokkos_ENABLE_CUDA=ON \
-DKokkosKernels_ENABLE_TPL_CUBLAS=ON \
-DKokkosKernels_INST_MEMSPACE_CUDAUVMSPACE=OFF

I get a section of the CMake output that looks like this:

=======================
KokkosKernels ETI Types
   Devices:  <Cuda,CudaSpace>;<Serial,HostSpace>
   Scalars:  double;float
   Ordinals: int
   Offsets:  size_t
   Layouts:  LayoutLeft

KokkosKernels TPLs
   CUBLAS:      /opt/nvidia/hpc_sdk/Linux_x86_64/21.11/profilers/Nsight_Compute/../../math_libs/11.5/lib64/libcublas.so
   CUSPARSE:    /opt/nvidia/hpc_sdk/Linux_x86_64/21.11/profilers/Nsight_Compute/../../math_libs/11.5/lib64/libcusparse.so
=======================

Does this work for you?

[sblondel]

I don't have access to Perlmutter myself, if @jczhang07 could try it that would be great.

@cwpearson where you able to reproduce our issue first or did you manage to build Kokkos Kernels immediately?

[sblondel]

I now have access to Perlmutter and tried compiling Kokkos Kernels with the same modules, same export variables, same options, but I still get the same error:

-- Cray Programming Environment 2.7.15 CXX
-- Setting policy CMP0074 to use <Package>_ROOT variables
-- The project name is: KokkosKernels
-- Enabled Kokkos devices: CUDA;SERIAL
CMake Warning at /global/homes/s/sblondel/perlmutter_install/kokkos_cuda/lib64/cmake/Kokkos/KokkosConfigCommon.cmake:34 (MESSAGE):
  The installed Kokkos configuration does not support CXX extensions.
  Forcing -DCMAKE_CXX_EXTENSIONS=Off
Call Stack (most recent call first):
  /global/homes/s/sblondel/perlmutter_install/kokkos_cuda/lib64/cmake/Kokkos/KokkosConfig.cmake:62 (INCLUDE)
  CMakeLists.txt:109 (FIND_PACKAGE)


-- Found Kokkos at /global/homes/s/sblondel/perlmutter_install/kokkos_cuda/lib64/cmake/Kokkos
-- Looking for C++ include pthread.h
-- Looking for C++ include pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Found CUDA: /opt/nvidia/hpc_sdk/Linux_x86_64/21.11/cuda/11.5 (found version "11.5") 
CMake Error at /global/common/software/nersc/pm-2021q4/sw/cmake-3.22.0/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find TPLCUBLAS (missing: CUDA_CUBLAS_LIBRARIES)
Call Stack (most recent call first):
  /global/common/software/nersc/pm-2021q4/sw/cmake-3.22.0/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  cmake/Modules/FindTPLCUBLAS.cmake:13 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  cmake/kokkoskernels_tpls.cmake:376 (FIND_PACKAGE)
  cmake/kokkoskernels_tpls.cmake:495 (KOKKOSKERNELS_IMPORT_TPL)
  CMakeLists.txt:196 (INCLUDE)

[jczhang07]

I still met the same error today with updated Perlmutter default settings.

[cwpearson]

I think this is a related PR: https://github.com/kokkos/kokkos-kernels/pull/2098 that fixes this issue for non-BLAS cuda libraries.

[cwpearson]

Probably this PR actually fixed this issue: https://github.com/kokkos/kokkos-kernels/pull/2075

[cwpearson]

Closing after communication on slack

[brian-kelley]

This has been fixed by #2075 and #2098 together - if you set CMake options like CUBLAS_ROOT=... or CUBLAS_LIBRARY_DIRS=... (similar for CUSPARSE/CUSOLVER) then the TPLs are detected correctly and shown in the CMake output.