Xingqiu Yuan

The github repo https://github.com/xyuan/e3sm_p3_shoc can reproduce this issue using the following script to create the run: ``` #!/bin/bash #E3SM=/ccs/home/yuanx/e3sm E3SM=/gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher OUTPUT=/gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS DATA=/gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS CASE_ROOT=${OUTPUT} #COMPSET=F2010-MMF1 COMPSET=F-MMFXX-P3 #COMPSET=F-MMFXX #RES=ne4pg2_ne4pg2 RES=ne30pg2_ne30pg2 #RES=ne120pg2_r05_oECv3 COMPILER=crayclanggpu...

> @xyuan or @sarats , could you also open an OLCF helpdesk ticket with this information? Since this is an interaction between GPFS and cray-mpich we'll probably need to pull...

I tried to use the following modules, [[email protected] F-MMFXX-P3.ne4pg2_ne4pg2.crusher.amdclanggpu.1x1]$ module list Currently Loaded Modules: 1) amd/5.1.0 5) xalt/1.3.0 9) craype-accel-amd-gfx90a 13) subversion/1.14.0 17) cray-libsci/21.08.1.2 2) craype/2.7.15 6) DefApps/default 10) rocm/5.1.0...

> Looks like we need to submit an AMD compiler bug. Can you give me permission to access https://github.com/xyuan/e3sm_p3_shoc/tree/e3sm_p3_shoc_hip? Github is telling me it doesn't exist. > > Or could...

@twhite-cray this is the script that I used to run MMF jobs using P3 ``` #!/bin/bash CASE_ROOT=$(pwd) #E3SM=/ccs/home/yuanx/e3sm E3SM=/gpfs/alpine/cli115/scratch/yuanx/e3sm_p3_crusher OUTPUT=/gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS DATA=/gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS #COMPSET=F2010-MMF1 COMPSET=F-MMFXX-P3 #COMPSET=F-MMFXX RES=ne4pg2_ne4pg2 COMPILER=crayclanggpu MACH=crusher PROJ=cli133 PELAYOUT=4x1 CASE=${COMPSET}.${RES}.${MACH}.${COMPILER}.${PELAYOUT}...

I copied the branch to world-shared, please check, for the github branch, please let me push some most recently change to it, it didn’t include machine configuration file for crusher...

I generated the cui files, and it is located here, [[email protected] shoc]$ pwd /gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS/F-MMFXX-P3.ne4pg2_ne4pg2.crusher.amdclanggpu.16x1.08/bld/cmake-bld/scream/src/physics/shoc [[email protected] shoc]$ ls -al total 5108 drwxr-xr-x 3 yuanx yuanx 4096 Jul 11 14:49 . drwxr-xr-x...

The following runscript for HiP test case is used, #!/bin/bash CASE_ROOT=$(pwd) E3SM=/ccs/home/yuanx/e3sm OUTPUT=/gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS DATA=/gpfs/alpine/cli115/scratch/yuanx/ACME_SIMULATIONS COMPSET=F2010-MMF1 #COMPSET=F-MMFXX-P3 RES=ne4pg2_ne4pg2 COMPILER=gnugpu MACH=crusher PROJ=cli133 PELAYOUT=1x1 CASE=${COMPSET}.${RES}.${MACH}.${COMPILER}.${PELAYOUT} echo echo ${CASE} echo ${E3SM}/cime/scripts/create_newcase -case ${CASE_ROOT}/${CASE} -compset...

This test branch can be used with the latest E3SM master, with some changes to add HiP support in the config_machine.xml, gnugpu_crusher.cmake, please copy it from my working branch

After setting up MPICH environment (LD_LIBRARY_PATH, and INCLUDE_PATH) correctly, it is still unable to find libmpi_gpu_91.so library in the mpich library directory at FindMPI.cmake test case