Add LLNL Tioga modules

[sbryngelson]

For LLNL Tioga MI250X nodes.

Though flux should be used, there are Slurm wrappers. One can get on a node via salloc -p pdebug -N 1 -t 01:00:00 and use srun later (with some minor changes, documented below).

Modules

  1) StdEnv             (S)   6) libfabric/2.1           11) cray-hdf5/1.12.2.11  16) craype-accel-amd-gfx90a
  2) craype-network-ofi       7) omniperf/2.0.0beta1     12) cray-mpich/8.1.27    17) cray-fftw/3.3.10.8
  3) cce/16.0.1               8) perftools-base/24.07.0  13) flux_wrappers/0.1
  4) cray-python/3.11.7       9) craype/2.7.32           14) rocm/5.5.1
  5) cmake/3.29.2            10) PrgEnv-cray/8.5.0       15) craype-x86-trento

which requires an unloading of cray-libsci for some reason.

We need the CMakeLists.txt patch for MI250X:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 5b72a285..b3dc497f 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -474,6 +474,7 @@ function(MFC_SETUP_TARGET)
         elseif(CMAKE_Fortran_COMPILER_ID STREQUAL "Cray")
             find_package(hipfort COMPONENTS hip CONFIG REQUIRED)
             target_link_libraries(${ARGS_TARGET} PRIVATE hipfort::hip hipfort::hipfort-amdgcn)
+            target_link_libraries(${ARGS_TARGET} PRIVATE stdc++)
         endif()
     elseif (CMAKE_Fortran_COMPILER_ID STREQUAL "Cray")
         target_compile_options(${ARGS_TARGET} PRIVATE "SHELL:-h noacc" "SHELL:-x ACC")

Tioga require s

export CC=cc
export FC=ftn
export CXX=CC

at build time

• Need to make sure that the cmake patch doesn't break Frontier cases
• The modules need to be added to ./mfc.sh load config
• For the computer mako file, we require the addition of LD_LIBRARY_PATH=$CRAY_LD_LIBRARY_PATH:$LD_LIBRARY_PATH after the MPICH_GPU statement. We also need = signs in srun, so:

       % if engine == 'interactive':
             --nodes=${nodes} --ntasks-per-node=${tasks_per_node} \
             --cpus-per-task=7                                    \
          % if gpu:
             --gpus-per-task=1 --gpu-bind=closest                 \

in toolchain/templates/tioga.mako

• Case-optimization is failing. I think this might be a known bug #566 on Frontier.

[sbryngelson]

@wilfonba this is yours now 😄

[wilfonba]

On my list. I need to get back to Tuo too...