E3SM-Project
add hacky atm tend diag
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:
Pull Request Auto Testing STARTING (click to expand)
Build Information
Test Name: SCREAM_PullRequest_Autotester_Mappy
- Build Num: 5337
- Status: STARTED
Jenkins Parameters
| Parameter Name | Value |
|---|---|
| PR_LABELS | diagnostic |
| PULLREQUESTNUM | 2810 |
| SCREAM_SOURCE_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_SOURCE_SHA | 033fad69efbc21b466a81415db93b78dc2173d97 |
| SCREAM_TARGET_BRANCH | master |
| SCREAM_TARGET_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_TARGET_SHA | a73d48a5f8556e5240b64b037bc60d42cb5f2413 |
| TEST_REPO_ALIAS | SCREAM |
Build Information
Test Name: SCREAM_PullRequest_Autotester_Weaver
- Build Num: 5655
- Status: STARTED
Jenkins Parameters
| Parameter Name | Value |
|---|---|
| PR_LABELS | diagnostic |
| PULLREQUESTNUM | 2810 |
| SCREAM_SOURCE_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_SOURCE_SHA | 033fad69efbc21b466a81415db93b78dc2173d97 |
| SCREAM_TARGET_BRANCH | master |
| SCREAM_TARGET_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_TARGET_SHA | a73d48a5f8556e5240b64b037bc60d42cb5f2413 |
| TEST_REPO_ALIAS | SCREAM |
Using Repos:
- Repo: SCREAM (E3SM-Project/scream)
- Branch: mahf708/diags/atm-tend
- SHA: 033fad69efbc21b466a81415db93b78dc2173d97
- Mode: TEST_REPO
Pull Request Author: mahf708
Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED
Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.
Pull Request Auto Testing has FAILED (click to expand)
Build Information
Test Name: SCREAM_PullRequest_Autotester_Mappy
- Build Num: 5337
- Status: FAILED
Jenkins Parameters
| Parameter Name | Value |
|---|---|
| PR_LABELS | diagnostic |
| PULLREQUESTNUM | 2810 |
| SCREAM_SOURCE_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_SOURCE_SHA | 033fad69efbc21b466a81415db93b78dc2173d97 |
| SCREAM_TARGET_BRANCH | master |
| SCREAM_TARGET_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_TARGET_SHA | a73d48a5f8556e5240b64b037bc60d42cb5f2413 |
| TEST_REPO_ALIAS | SCREAM |
Build Information
Test Name: SCREAM_PullRequest_Autotester_Weaver
- Build Num: 5655
- Status: FAILED
Jenkins Parameters
| Parameter Name | Value |
|---|---|
| PR_LABELS | diagnostic |
| PULLREQUESTNUM | 2810 |
| SCREAM_SOURCE_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_SOURCE_SHA | 033fad69efbc21b466a81415db93b78dc2173d97 |
| SCREAM_TARGET_BRANCH | master |
| SCREAM_TARGET_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_TARGET_SHA | a73d48a5f8556e5240b64b037bc60d42cb5f2413 |
| TEST_REPO_ALIAS | SCREAM |
SCREAM_PullRequest_Autotester_Mappy # 5337 FAILED (click to see last 100 lines of console output)
Finished SETUP for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 in 2.057482 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 with 1 procs
Finished SETUP for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 in 2.532753 seconds (PASS)
Starting SHAREDLIB_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 with 1 procs
Finished SHAREDLIB_BUILD for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics in 55.816016 seconds (PASS)
Starting MODEL_BUILD for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics with 16 procs
Finished SHAREDLIB_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 in 54.624913 seconds (PASS)
Starting MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 with 16 procs
Finished SHAREDLIB_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 in 54.636400 seconds (PASS)
Starting MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 with 16 procs
Finished SHAREDLIB_BUILD for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu in 71.702469 seconds (PASS)
Starting MODEL_BUILD for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu with 16 procs
Finished SHAREDLIB_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 in 69.329909 seconds (PASS)
Finished SHAREDLIB_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 84.561902 seconds (PASS)
Finished SHAREDLIB_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 170.103301 seconds (PASS)
Finished SHAREDLIB_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 in 193.972466 seconds (PASS)
Starting MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 with 16 procs
Finished MODEL_BUILD for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics in 173.028117 seconds (PASS)
Starting RUN for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics in 1.591888 seconds (PEND). [COMPLETED 1 of 8]
Starting MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 16 procs
Finished MODEL_BUILD for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 in 175.854049 seconds (PASS)
Starting RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 in 0.947126 seconds (PEND). [COMPLETED 2 of 8]
Starting MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 with 16 procs
Finished MODEL_BUILD for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 in 177.133335 seconds (PASS)
Starting RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 in 0.980685 seconds (PEND). [COMPLETED 3 of 8]
Starting MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 16 procs
Finished MODEL_BUILD for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu in 321.685689 seconds (PASS)
Starting RUN for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu with 1 proc on interactive node and 16 procs on compute nodes
Finished RUN for test ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu in 1.439393 seconds (PEND). [COMPLETED 4 of 8]
Finished MODEL_BUILD for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 in 301.809154 seconds (PASS)
Starting RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 in 1.063732 seconds (PEND). [COMPLETED 5 of 8]
Finished MODEL_BUILD for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 351.522461 seconds (PASS)
Starting RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 1.029960 seconds (PEND). [COMPLETED 6 of 8]
Finished MODEL_BUILD for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 458.446887 seconds (PASS)
Starting RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 in 1.800316 seconds (PEND). [COMPLETED 7 of 8]
Finished MODEL_BUILD for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 in 530.745416 seconds (PASS)
Starting RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 with 1 proc on interactive node and 64 procs on compute nodes
Finished RUN for test ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 in 0.949966 seconds (PEND). [COMPLETED 8 of 8]
Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240503_172857_ao85yf
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 RUN
Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240503_172857_ao85yf
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 RUN
Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240503_172857_ao85yf
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 RUN
Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240503_172857_ao85yf
PASS ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu RUN
Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240503_172857_ao85yf
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240503_172857_ao85yf
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240503_172857_ao85yf
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 RUN
Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240503_172857_ao85yf
test-scheduler took 803.8045818805695 seconds'
+ [[ 0 != 0 ]]
+ set +x
######################################################
FAILS DETECTED:
SCREAM STANDALONE TESTING FAILED!
Build type full_sp_debug failed at testing time. Here's a list of failed tests:
233:atm_tend
Error(s) occurred during test phase
OVERALL STATUS: FAIL
Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5337/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true && ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m mappy
RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5337/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export PATH=/ascldap/users/jgfouca/packages/valgrind-3.22.0/bin:$PATH && export OMP_PROC_BIND=spread && true && ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m mappy
FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5337/scream/components/eamxx
mappy failed
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 109178 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins2963938717347835252.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: [email protected]
Finished: FAILURE
SCREAM_PullRequest_Autotester_Weaver # 5655 FAILED (click to see last 100 lines of console output)
CMake Error at /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5655/scream/components/eamxx/cmake/ctest_script.cmake:76 (message):
Test had fails
===============================================================================
Testing '''033fad69efbc21b466a81415db93b78dc2173d97''' for test '''release'''
RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5655/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5655/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5655/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5655/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5655/scream/components/eamxx/ctest-build/release
Testing '''033fad69efbc21b466a81415db93b78dc2173d97''' for test '''full_sp_debug'''
RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5655/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5655/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5655/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5655/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug" '''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5655/scream/components/eamxx/ctest-build/full_sp_debug
Testing '''033fad69efbc21b466a81415db93b78dc2173d97''' for test '''full_debug'''
RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5655/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5655/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5655/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5655/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5655/scream/components/eamxx/ctest-build/full_debug
Build type full_debug failed at testing time. Here'''s a list of failed tests:
73:atm_tend
Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:
68:atm_tend
Build type release failed at testing time. Here'''s a list of failed tests:
73:atm_tend
Error(s) occurred during test phase
OVERALL STATUS: FAIL
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5655/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5655/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5655/scream/components/eamxx
weaver failed'
- errors='Build type full_debug failed at testing time. Here'''s a list of failed tests:
73:atm_tend
Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:
68:atm_tend
Build type release failed at testing time. Here'''s a list of failed tests:
73:atm_tend
Error(s) occurred during test phase
OVERALL STATUS: FAIL
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5655/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5655/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5655/scream/components/eamxx
weaver failed'
- SA_FAILURES_DETAILS+='Build type full_debug failed at testing time. Here'''s a list of failed tests:
73:atm_tend
Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:
68:atm_tend
Build type release failed at testing time. Here'''s a list of failed tests:
73:atm_tend
Error(s) occurred during test phase
OVERALL STATUS: FAIL
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5655/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5655/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5655/scream/components/eamxx
weaver failed'
- [[ 1 == 0 ]]
- [[ weaver == \m\a\p\p\y ]]
- set +x
######################################################
FAILS DETECTED:
SCREAM STANDALONE TESTING FAILED!
Build type full_debug failed at testing time. Here's a list of failed tests:
73:atm_tend
Build type full_sp_debug failed at testing time. Here's a list of failed tests:
68:atm_tend
Build type release failed at testing time. Here's a list of failed tests:
73:atm_tend
Error(s) occurred during test phase
OVERALL STATUS: FAIL
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5655/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5655/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5655/scream/components/eamxx
weaver failed
######################################################
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins11560560000894107173.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: [email protected]
Finished: FAILURE
Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:
Pull Request Auto Testing STARTING (click to expand)
Build Information
Test Name: SCREAM_PullRequest_Autotester_Mappy
- Build Num: 5339
- Status: STARTED
Jenkins Parameters
| Parameter Name | Value |
|---|---|
| PR_LABELS | AT: RETEST;diagnostic |
| PULLREQUESTNUM | 2810 |
| SCREAM_SOURCE_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_SOURCE_SHA | 9b14fd1fa55ccb82b548be4e827d0257c92492e1 |
| SCREAM_TARGET_BRANCH | master |
| SCREAM_TARGET_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_TARGET_SHA | a73d48a5f8556e5240b64b037bc60d42cb5f2413 |
| TEST_REPO_ALIAS | SCREAM |
Build Information
Test Name: SCREAM_PullRequest_Autotester_Weaver
- Build Num: 5656
- Status: STARTED
Jenkins Parameters
| Parameter Name | Value |
|---|---|
| PR_LABELS | AT: RETEST;diagnostic |
| PULLREQUESTNUM | 2810 |
| SCREAM_SOURCE_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_SOURCE_SHA | 9b14fd1fa55ccb82b548be4e827d0257c92492e1 |
| SCREAM_TARGET_BRANCH | master |
| SCREAM_TARGET_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_TARGET_SHA | a73d48a5f8556e5240b64b037bc60d42cb5f2413 |
| TEST_REPO_ALIAS | SCREAM |
Using Repos:
- Repo: SCREAM (E3SM-Project/scream)
- Branch: mahf708/diags/atm-tend
- SHA: 9b14fd1fa55ccb82b548be4e827d0257c92492e1
- Mode: TEST_REPO
Pull Request Author: mahf708
Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED
Pull Request Auto Testing has PASSED (click to expand)
Build Information
Test Name: SCREAM_PullRequest_Autotester_Mappy
- Build Num: 5339
- Status: PASSED
Jenkins Parameters
| Parameter Name | Value |
|---|---|
| PR_LABELS | AT: RETEST;diagnostic |
| PULLREQUESTNUM | 2810 |
| SCREAM_SOURCE_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_SOURCE_SHA | 9b14fd1fa55ccb82b548be4e827d0257c92492e1 |
| SCREAM_TARGET_BRANCH | master |
| SCREAM_TARGET_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_TARGET_SHA | a73d48a5f8556e5240b64b037bc60d42cb5f2413 |
| TEST_REPO_ALIAS | SCREAM |
Build Information
Test Name: SCREAM_PullRequest_Autotester_Weaver
- Build Num: 5656
- Status: PASSED
Jenkins Parameters
| Parameter Name | Value |
|---|---|
| PR_LABELS | AT: RETEST;diagnostic |
| PULLREQUESTNUM | 2810 |
| SCREAM_SOURCE_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_SOURCE_SHA | 9b14fd1fa55ccb82b548be4e827d0257c92492e1 |
| SCREAM_TARGET_BRANCH | master |
| SCREAM_TARGET_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_TARGET_SHA | a73d48a5f8556e5240b64b037bc60d42cb5f2413 |
| TEST_REPO_ALIAS | SCREAM |
Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging THE LAST COMMIT TO THIS PULL REQUEST HAS NOT BEEN REVIEWED YET!
All Jobs Finished; status = PASSED, target_sha=a73d48a5f8556e5240b64b037bc60d42cb5f2413, However Inspection must be performed before merge can occur...
Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging THE LAST COMMIT TO THIS PULL REQUEST HAS BEEN REVIEWED, BUT NOT ACCEPTED OR REQUIRES CHANGES!
All Jobs Finished; status = PASSED, target_sha=a73d48a5f8556e5240b64b037bc60d42cb5f2413, However Inspection must be performed before merge can occur...
All Jobs Finished; status = PASSED, target_sha=a73d48a5f8556e5240b64b037bc60d42cb5f2413, However Inspection must be performed before merge can occur...
WIP status: I will try to merge this after upstream merge to bring kokkos 4.2. The PR doesn't need kokkos 4.2, but I can edit some parts in it to take advantage of kokkos 4.2
@bartgol this PR is ready, three items for your input, please :) it's been only a few weeks, but I am 100% rusty on details, so please bear with me
- Do you agree/disagree the last test is a bad idea? Any more suggestions for more tests?
- Do you think the units are handled fine? Do I need to address the TODO item in the code? I think your PRs in EKAT made that automated but I can't be so sure
- Do you see any potential problem with the logic here? Note that I am relying on something I didn't like (technically calculating the difference in the fields between end and start of step, but the dt is calculated as difference between end and end --- this is fine because it yields the same results, but I didn't like this, but I couldn't get it to work; ideally, I wanted both differencing to be on the same exact time spans)
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:
Pull Request Auto Testing STARTING (click to expand)
Build Information
Test Name: SCREAM_PullRequest_Autotester_Mappy
- Build Num: 5614
- Status: STARTED
Jenkins Parameters
| Parameter Name | Value |
|---|---|
| PR_LABELS | AT: AUTOMERGE;priority:low;diagnostic |
| PULLREQUESTNUM | 2810 |
| SCREAM_SOURCE_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_SOURCE_SHA | 86586bf98119316f9a45f6e5092a391a6d87e98e |
| SCREAM_TARGET_BRANCH | master |
| SCREAM_TARGET_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_TARGET_SHA | 33144527e141874511f91287520df70d887ee672 |
| TEST_REPO_ALIAS | SCREAM |
Build Information
Test Name: SCREAM_PullRequest_Autotester_Weaver
- Build Num: 5861
- Status: STARTED
Jenkins Parameters
| Parameter Name | Value |
|---|---|
| PR_LABELS | AT: AUTOMERGE;priority:low;diagnostic |
| PULLREQUESTNUM | 2810 |
| SCREAM_SOURCE_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_SOURCE_SHA | 86586bf98119316f9a45f6e5092a391a6d87e98e |
| SCREAM_TARGET_BRANCH | master |
| SCREAM_TARGET_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_TARGET_SHA | 33144527e141874511f91287520df70d887ee672 |
| TEST_REPO_ALIAS | SCREAM |
Using Repos:
- Repo: SCREAM (E3SM-Project/scream)
- Branch: mahf708/diags/atm-tend
- SHA: 86586bf98119316f9a45f6e5092a391a6d87e98e
- Mode: TEST_REPO
Pull Request Author: mahf708
Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED
Pull Request Auto Testing has PASSED (click to expand)
Build Information
Test Name: SCREAM_PullRequest_Autotester_Mappy
- Build Num: 5614
- Status: PASSED
Jenkins Parameters
| Parameter Name | Value |
|---|---|
| PR_LABELS | AT: AUTOMERGE;priority:low;diagnostic |
| PULLREQUESTNUM | 2810 |
| SCREAM_SOURCE_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_SOURCE_SHA | 86586bf98119316f9a45f6e5092a391a6d87e98e |
| SCREAM_TARGET_BRANCH | master |
| SCREAM_TARGET_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_TARGET_SHA | 33144527e141874511f91287520df70d887ee672 |
| TEST_REPO_ALIAS | SCREAM |
Build Information
Test Name: SCREAM_PullRequest_Autotester_Weaver
- Build Num: 5861
- Status: PASSED
Jenkins Parameters
| Parameter Name | Value |
|---|---|
| PR_LABELS | AT: AUTOMERGE;priority:low;diagnostic |
| PULLREQUESTNUM | 2810 |
| SCREAM_SOURCE_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_SOURCE_SHA | 86586bf98119316f9a45f6e5092a391a6d87e98e |
| SCREAM_TARGET_BRANCH | master |
| SCREAM_TARGET_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_TARGET_SHA | 33144527e141874511f91287520df70d887ee672 |
| TEST_REPO_ALIAS | SCREAM |
Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging THE LAST COMMIT TO THIS PULL REQUEST HAS NOT BEEN REVIEWED YET!
All Jobs Finished; status = PASSED, target_sha=33144527e141874511f91287520df70d887ee672, However Inspection must be performed before merge can occur...
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:
Pull Request Auto Testing STARTING (click to expand)
Build Information
Test Name: SCREAM_PullRequest_Autotester_Mappy
- Build Num: 5615
- Status: STARTED
Jenkins Parameters
| Parameter Name | Value |
|---|---|
| PR_LABELS | AT: AUTOMERGE;priority:low;diagnostic |
| PULLREQUESTNUM | 2810 |
| SCREAM_SOURCE_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_SOURCE_SHA | 613a260d59f98c4075a43f2bc319f7f13e4fff8e |
| SCREAM_TARGET_BRANCH | master |
| SCREAM_TARGET_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_TARGET_SHA | 33144527e141874511f91287520df70d887ee672 |
| TEST_REPO_ALIAS | SCREAM |
Build Information
Test Name: SCREAM_PullRequest_Autotester_Weaver
- Build Num: 5862
- Status: STARTED
Jenkins Parameters
| Parameter Name | Value |
|---|---|
| PR_LABELS | AT: AUTOMERGE;priority:low;diagnostic |
| PULLREQUESTNUM | 2810 |
| SCREAM_SOURCE_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_SOURCE_SHA | 613a260d59f98c4075a43f2bc319f7f13e4fff8e |
| SCREAM_TARGET_BRANCH | master |
| SCREAM_TARGET_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_TARGET_SHA | 33144527e141874511f91287520df70d887ee672 |
| TEST_REPO_ALIAS | SCREAM |
Using Repos:
- Repo: SCREAM (E3SM-Project/scream)
- Branch: mahf708/diags/atm-tend
- SHA: 613a260d59f98c4075a43f2bc319f7f13e4fff8e
- Mode: TEST_REPO
Pull Request Author: mahf708
Status Flag 'Pull Request AutoTester' - Error: Jenkins Jobs - A user has pushed a change to the PR before testing completed. NEW EVENT 'committed', ID C_kwDOCEfuetoAKGExNjIxZGY0ZmFiNTYxZDI2MmIwYjBjNTkyYTFjNjlkNDNhMGY5YmU... The Jenkins Jobs will be shutdown; Testing of this PR must occur again.
Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED
Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.
Pull Request Auto Testing has FAILED (click to expand)
Build Information
Test Name: SCREAM_PullRequest_Autotester_Mappy
- Build Num: 5615
- Status: ERROR
Jenkins Parameters
| Parameter Name | Value |
|---|---|
| PR_LABELS | AT: AUTOMERGE;priority:low;diagnostic |
| PULLREQUESTNUM | 2810 |
| SCREAM_SOURCE_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_SOURCE_SHA | 613a260d59f98c4075a43f2bc319f7f13e4fff8e |
| SCREAM_TARGET_BRANCH | master |
| SCREAM_TARGET_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_TARGET_SHA | 33144527e141874511f91287520df70d887ee672 |
| TEST_REPO_ALIAS | SCREAM |
Build Information
Test Name: SCREAM_PullRequest_Autotester_Weaver
- Build Num: 5862
- Status: ERROR
Jenkins Parameters
| Parameter Name | Value |
|---|---|
| PR_LABELS | AT: AUTOMERGE;priority:low;diagnostic |
| PULLREQUESTNUM | 2810 |
| SCREAM_SOURCE_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_SOURCE_SHA | 613a260d59f98c4075a43f2bc319f7f13e4fff8e |
| SCREAM_TARGET_BRANCH | master |
| SCREAM_TARGET_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_TARGET_SHA | 33144527e141874511f91287520df70d887ee672 |
| TEST_REPO_ALIAS | SCREAM |
SCREAM_PullRequest_Autotester_Mappy # 5615 ERROR (click to see last 100 lines of console output)
> git config -f .gitmodules --get submodule.components/mpas-ocean/src/cvmix.path # timeout=10
> git config --get submodule.components/mpas-ocean/src/BGC.url # timeout=10
> git remote # timeout=10
> git config --get remote.origin.url # timeout=10
> git config -f .gitmodules --get submodule.components/mpas-ocean/src/BGC.path # timeout=10
> git config --get submodule.MARBL.url # timeout=10
> git remote # timeout=10
> git config --get remote.origin.url # timeout=10
> git config -f .gitmodules --get submodule.MARBL.path # timeout=10
> git config --get submodule.components/mpas-ocean/src/gotm.url # timeout=10
> git remote # timeout=10
> git config --get remote.origin.url # timeout=10
> git config -f .gitmodules --get submodule.components/mpas-ocean/src/gotm.path # timeout=10
> git config --get submodule.externals/mct.url # timeout=10
> git remote # timeout=10
> git config --get remote.origin.url # timeout=10
> git config -f .gitmodules --get submodule.externals/mct.path # timeout=10
> git config --get submodule.components/mpas-albany-landice/src/SeaLevelModel.url # timeout=10
> git remote # timeout=10
> git config --get remote.origin.url # timeout=10
> git config -f .gitmodules --get submodule.components/mpas-albany-landice/src/SeaLevelModel.path # timeout=10
> git config --get submodule.components/mpas-ocean/src/SHTNS.url # timeout=10
> git remote # timeout=10
> git config --get remote.origin.url # timeout=10
> git config -f .gitmodules --get submodule.components/mpas-ocean/src/SHTNS.path # timeout=10
> git config --get submodule.components/mpas-ocean/src/FFTW.url # timeout=10
> git remote # timeout=10
> git config --get remote.origin.url # timeout=10
> git config -f .gitmodules --get submodule.components/mpas-ocean/src/FFTW.path # timeout=10
> git config --get submodule.components/mpas-ocean/src/ppr.url # timeout=10
> git remote # timeout=10
> git config --get remote.origin.url # timeout=10
> git config -f .gitmodules --get submodule.components/mpas-ocean/src/ppr.path # timeout=10
> git config --get submodule.components/ww3/src/source/WW3.url # timeout=10
> git remote # timeout=10
> git config --get remote.origin.url # timeout=10
> git config -f .gitmodules --get submodule.components/ww3/src/source/WW3.path # timeout=10
> git config --get submodule.components/eam/src/physics/crm/pam/external.url # timeout=10
> git remote # timeout=10
> git config --get remote.origin.url # timeout=10
> git config -f .gitmodules --get submodule.components/eam/src/physics/crm/pam/external.path # timeout=10
> git config --get submodule.components/mpas-seaice/src/icepack.url # timeout=10
> git remote # timeout=10
> git config --get remote.origin.url # timeout=10
> git config -f .gitmodules --get submodule.components/mpas-seaice/src/icepack.path # timeout=10
> git config --get submodule.externals/haero.url # timeout=10
> git remote # timeout=10
> git config --get remote.origin.url # timeout=10
> git config -f .gitmodules --get submodule.externals/haero.path # timeout=10
> git config --get submodule.externals/mam4xx.url # timeout=10
> git remote # timeout=10
> git config --get remote.origin.url # timeout=10
> git config -f .gitmodules --get submodule.externals/mam4xx.path # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
> git submodule update --init --recursive components/elm/src/external_models/fates # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
> git submodule update --init --recursive components/elm/src/external_models/mpp # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
> git submodule update --init --recursive components/elm/src/external_models/sbetr # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
> git submodule update --init --recursive components/eam/src/physics/rrtmgp/external # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
> git submodule update --init --recursive externals/scorpio # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
> git submodule update --init --recursive externals/scorpio_classic # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
> git submodule update --init --recursive components/eam/src/physics/cosp2/external # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
> git submodule update --init --recursive cime # timeout=10
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 54122 killed;
[ssh-agent] Stopped.
Build was aborted
Aborted by Luca Bertagna
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins9534970632582758072.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Finished: ABORTED
SCREAM_PullRequest_Autotester_Weaver # 5862 ERROR (click to see last 100 lines of console output)
> /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.externals/scorpio_classic.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.externals/scorpio_classic.path # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.cosp2.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.cosp2.path # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.cime.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.cime.path # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.externals/ekat.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.externals/ekat.path # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.externals/YAKL.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.externals/YAKL.path # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.externals/cub.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.externals/cub.path # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.components/mpas-ocean/src/cvmix.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.components/mpas-ocean/src/cvmix.path # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.components/mpas-ocean/src/BGC.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.components/mpas-ocean/src/BGC.path # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.MARBL.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.MARBL.path # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.components/mpas-ocean/src/gotm.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.components/mpas-ocean/src/gotm.path # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.externals/mct.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.externals/mct.path # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.components/mpas-albany-landice/src/SeaLevelModel.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.components/mpas-albany-landice/src/SeaLevelModel.path # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.components/mpas-ocean/src/SHTNS.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.components/mpas-ocean/src/SHTNS.path # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.components/mpas-ocean/src/FFTW.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.components/mpas-ocean/src/FFTW.path # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.components/mpas-ocean/src/ppr.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.components/mpas-ocean/src/ppr.path # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.components/ww3/src/source/WW3.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.components/ww3/src/source/WW3.path # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.components/eam/src/physics/crm/pam/external.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.components/eam/src/physics/crm/pam/external.path # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.components/mpas-seaice/src/icepack.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.components/mpas-seaice/src/icepack.path # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.externals/haero.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.externals/haero.path # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get submodule.externals/mam4xx.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git remote # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config --get remote.origin.url # timeout=10
> /home/projects/ppc64le/git/2.10.1/bin/git config -f .gitmodules --get submodule.externals/mam4xx.path # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
> /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive components/elm/src/external_models/fates # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
> /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive components/elm/src/external_models/mpp # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
> /home/projects/ppc64le/git/2.10.1/bin/git submodule update --init --recursive components/elm/src/external_models/sbetr # timeout=10
Build was aborted
Aborted by Luca Bertagna
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins11062768091656398500.sh
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:
Pull Request Auto Testing STARTING (click to expand)
Build Information
Test Name: SCREAM_PullRequest_Autotester_Mappy
- Build Num: 5616
- Status: STARTED
Jenkins Parameters
| Parameter Name | Value |
|---|---|
| PR_LABELS | AT: AUTOMERGE;priority:low;diagnostic |
| PULLREQUESTNUM | 2810 |
| SCREAM_SOURCE_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_SOURCE_SHA | a1621df4fab561d262b0b0c592a1c69d43a0f9be |
| SCREAM_TARGET_BRANCH | master |
| SCREAM_TARGET_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_TARGET_SHA | 33144527e141874511f91287520df70d887ee672 |
| TEST_REPO_ALIAS | SCREAM |
Build Information
Test Name: SCREAM_PullRequest_Autotester_Weaver
- Build Num: 5863
- Status: STARTED
Jenkins Parameters
| Parameter Name | Value |
|---|---|
| PR_LABELS | AT: AUTOMERGE;priority:low;diagnostic |
| PULLREQUESTNUM | 2810 |
| SCREAM_SOURCE_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_SOURCE_SHA | a1621df4fab561d262b0b0c592a1c69d43a0f9be |
| SCREAM_TARGET_BRANCH | master |
| SCREAM_TARGET_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_TARGET_SHA | 33144527e141874511f91287520df70d887ee672 |
| TEST_REPO_ALIAS | SCREAM |
Using Repos:
- Repo: SCREAM (E3SM-Project/scream)
- Branch: mahf708/diags/atm-tend
- SHA: a1621df4fab561d262b0b0c592a1c69d43a0f9be
- Mode: TEST_REPO
Pull Request Author: mahf708
Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED
Pull Request Auto Testing has PASSED (click to expand)
Build Information
Test Name: SCREAM_PullRequest_Autotester_Mappy
- Build Num: 5616
- Status: PASSED
Jenkins Parameters
| Parameter Name | Value |
|---|---|
| PR_LABELS | AT: AUTOMERGE;priority:low;diagnostic |
| PULLREQUESTNUM | 2810 |
| SCREAM_SOURCE_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_SOURCE_SHA | a1621df4fab561d262b0b0c592a1c69d43a0f9be |
| SCREAM_TARGET_BRANCH | master |
| SCREAM_TARGET_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_TARGET_SHA | 33144527e141874511f91287520df70d887ee672 |
| TEST_REPO_ALIAS | SCREAM |
Build Information
Test Name: SCREAM_PullRequest_Autotester_Weaver
- Build Num: 5863
- Status: PASSED
Jenkins Parameters
| Parameter Name | Value |
|---|---|
| PR_LABELS | AT: AUTOMERGE;priority:low;diagnostic |
| PULLREQUESTNUM | 2810 |
| SCREAM_SOURCE_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_SOURCE_SHA | a1621df4fab561d262b0b0c592a1c69d43a0f9be |
| SCREAM_TARGET_BRANCH | master |
| SCREAM_TARGET_REPO | https://github.com/E3SM-Project/scream |
| SCREAM_TARGET_SHA | 33144527e141874511f91287520df70d887ee672 |
| TEST_REPO_ALIAS | SCREAM |
Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging THE LAST COMMIT TO THIS PULL REQUEST HAS NOT BEEN REVIEWED YET!
All Jobs Finished; status = PASSED, target_sha=33144527e141874511f91287520df70d887ee672, However Inspection must be performed before merge can occur...