Kokkos/Sacado CUDA configure problem

[isaki001]

sacado @etphipp

Hello,

I'm trying to build Trilinos with Kokkos/Sacado enabled, and CUDA as the backend but I am getting an error about an invalid CUDA compiler. I am using cmake 3.20.2 instead of 3.23.0. I had to edit the CMakeLists.txt at the top level to decrease the cmake minimum requirement. I'm not sure if that is what is getting me into trouble but I was hoping it would ok since cmake 3.17.1 was mentioned as a requirement in this link (https://github.com/trilinos/Trilinos/blob/master/INSTALL.rst#requirements)

Error

CMake Error at packages/kokkos/cmake/kokkos_test_cxx_std.cmake:132 (MESSAGE):
Invalid compiler for CUDA.  The compiler must be nvcc_wrapper or Clang or
NVC++ or use kokkos_launch_compiler, but compiler ID was GNU

CMake Configuration

I used the following command:

  cmake  ${TRILINOS_SOURCE_DIR} \
  -DCMAKE_INSTALL_PREFIX="." \
  -DCMAKE_CXX_COMPILER="${TRILINOS_SOURCE_DIR}/packages/kokkos/bin/nvcc_wrapper" \
  -DTrilinos_ENABLE_SACADO=ON \
  -DTrilinos_ENABLE_TESTS=ON  \
  -DTrilinos_ENABLE_Kokkos=ON  \
  -DTPL_ENABLE_CUDA=ON \
  -DKokkos_ARCH_VOLTA70=ON 

Unfortunately CMAKE_CXX_COMPILER is always skipped and GNU 8.5.0 is picked up as the C++ compiler:

-- USE_XSDK_DEFAULTS='FALSE'
-- BUILD_SHARED_LIBS='FALSE'
-- Setting CMAKE_BUILD_TYPE=RELEASE since it was not set ...
-- CMAKE_BUILD_TYPE='RELEASE'
-- The C compiler identification is GNU 8.5.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- CMAKE_C_COMPILER_ID='GNU'
-- CMAKE_C_COMPILER_VERSION='8.5.0'
-- The CXX compiler identification is GNU 8.5.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/isaki001/Trilinos/packages/kokkos/bin/nvcc_wrapper - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- CMAKE_CXX_COMPILER_ID='GNU'
-- CMAKE_CXX_COMPILER_VERSION='8.5.0'
-- The Fortran compiler identification is GNU 8.5.0

I tried setting the CXX environment variable instead but I was getting the same error. I had a separate installation of Kokkos as well and tried to pass the nvcc_wrapper from that installation to CMAKE_CXX_COMPILER but it didn't change anything.

Am I doing something wrong with my cmake command? Is there an example for building Trilinos with Kokkos that could use?

Environment

My installation directory is not under TRILINOS_SOURCE_DIR.

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_Mar__8_18:18:20_PST_2022
Cuda compilation tools, release 11.6, V11.6.124
Build cuda_11.6.r11.6/compiler.31057947_0

g++ --version
g++ (GCC) 8.5.0 20210514 (Red Hat 8.5.0-13)

cmake --version
cmake version 3.20.2

[etphipp]

The cmake version is not likely the source of your problems, but you may run into other issues later on. Did you perhaps run CMake using g++ as the compiler but then forget to clear the CMake cache? We usually put this in our scripts before running CMake:

rm -f CMakeCache.txt;
rm -rf CMakeFiles

as things like the compiler are cached from the previous run.

[isaki001]

I always cleared all the cache files so I don't think that's it.

I will see if I can get access to a newer CMake to be certain, but I don't know anything else to try.

I don't believe I'm missing any other requirements though. Should I be looking into any other issues from my environment?

[jhux2]

@bartlettroscoe

[bartlettroscoe]

@trilinos/framework, @sebrowne

[etphipp]

Actually I just looked a little closer at your output and it is picking up nvcc_wrapper as it should

-- Check for working CXX compiler: /home/isaki001/Trilinos/packages/kokkos/bin/nvcc_wrapper - skipped

I suspect there is something else going wrong with your compiler (probably a linking issue). If you look at CMakeOutput.log and CMakeError.log (which I believe are in the CMakeFiles directory), it should show what is happening when it tries to test the compiler can compile Cuda code.

[sebrowne]

We should definitely change the INSTALL.rst file to ensure that it aligns with cmake_minimum_required() (or remove it since it's a bit duplicative).

Just to clarify: You've tried -DCMAKE_CXX_COMPILER=/path/to/nvcc_wrapper, and let it run until it errors out?

[jhux2]

We should definitely change the INSTALL.rst file to ensure that it aligns with cmake_minimum_required() (or remove it since it's a bit duplicative).

I don't know of another resource like the INSTALL.rst for new/casual users. (Can one configure/build Trilinos casually?) So I would vote for updating rather than removing.

[bartlettroscoe]

We should definitely change the INSTALL.rst file to ensure that it aligns with cmake_minimum_required() (or remove it since it's a bit duplicative).

See PR #11625.

(Can one configure/build Trilinos casually?)

Sure, if you configure without CUDA or MPI and only enable Teuchos with BLAS, LAPACK and GCC all in the system path then you can configure Trilinos with:

$ cmake -DCMAKE_C_COMPILER=gcc -DTrilinos_ENABLE_Teuchos=ON <trilinosDir>

it should configure right out of the box. .😊

[isaki001]

Just to clarify: You've tried -DCMAKE_CXX_COMPILER=/path/to/nvcc_wrapper, and let it run until it errors out?

Yes, that's exactly what I did. I could not understand why it was being skipped once I run cmake. I'll look into CMakeOutput.log and CMakeError.log.

[github-actions[bot]]

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity. If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label. If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE. If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.

[github-actions[bot]]

This issue was closed due to inactivity for 395 days.