scream icon indicating copy to clipboard operation
scream copied to clipboard
verified publisher icon E3SM-Project

Fix a couple of issues in history restart

Open bartgol opened this issue 1 year ago • 6 comments

Namely:

  • Avoid opening the last output file if it's deemed full
  • When resuming an output file, reset the time dim length to the correct value (stored in rhist file). This allows to overwrite any snapshot that was written to file after rhist was written.

I was able to reproduce the issue in master by running the model_restart test with:

  • output every step
  • max 2 snaps/file
  • instant output (includes t=0 output)
  • run for 2 steps, then restart. Running the 2nd run (after restart) twice, would cause the shifted file to appear. With this branch, the generated files are the expected ones: t=0steps (2snaps, for t=0,1), t=2steps (2snaps, for t=2,3), t=4steps (1snap. for t=4).

Fixes #2889 Fixes #2892

bartgol avatar Jul 30 '24 23:07 bartgol

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5737
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;I/O;AT: AUTOMERGE;bugfix
PULLREQUESTNUM 2929
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 0bdead86c0f3a3b240d92efd3076a62cdc67dd65
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d508361a8e53f672662912e6fa02e8074703029
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5968
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;I/O;AT: AUTOMERGE;bugfix
PULLREQUESTNUM 2929
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 0bdead86c0f3a3b240d92efd3076a62cdc67dd65
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d508361a8e53f672662912e6fa02e8074703029
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: bartgol/eamxx/hist-restart-fix
  • SHA: 0bdead86c0f3a3b240d92efd3076a62cdc67dd65
  • Mode: TEST_REPO

Pull Request Author: bartgol

E3SM-Bot avatar Jul 30 '24 23:07 E3SM-Bot

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5737
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;I/O;AT: AUTOMERGE;bugfix
PULLREQUESTNUM 2929
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 0bdead86c0f3a3b240d92efd3076a62cdc67dd65
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d508361a8e53f672662912e6fa02e8074703029
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5968
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;I/O;AT: AUTOMERGE;bugfix
PULLREQUESTNUM 2929
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 0bdead86c0f3a3b240d92efd3076a62cdc67dd65
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d508361a8e53f672662912e6fa02e8074703029
TEST_REPO_ALIAS SCREAM
SCREAM_PullRequest_Autotester_Mappy # 5737 FAILED (click to see last 100 lines of console output) 

PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci.C.20240730_181332_ov2k5e
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240730_181332_ov2k5e
DIFF SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav.C.20240730_181332_ov2k5e
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240730_181332_ov2k5e
test-scheduler took 981.496740102768 seconds'
+ V1_FAILURES_DETAILS+='Waiting for tests to finish
FAIL ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240730_181332_ov2k5e
FAIL ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240730_181332_ov2k5e
FAIL ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240730_181332_ov2k5e
FAIL ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240730_181332_ov2k5e
PASS ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240730_181332_ov2k5e
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240730_181332_ov2k5e
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci.C.20240730_181332_ov2k5e
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240730_181332_ov2k5e
DIFF SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav.C.20240730_181332_ov2k5e
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240730_181332_ov2k5e
test-scheduler took 981.496740102768 seconds'
+ set +x
######################################################
FAILS DETECTED:
  SCREAM STANDALONE TESTING FAILED!
Build type full_debug failed at testing time. Here's a list of failed tests:
30:output_restart_np1
31:output_restart_np2
32:output_restart_np3
33:output_restart_np4
34:output_restart_check_INSTANT_np1
35:output_restart_check_INSTANT_np2
36:output_restart_check_INSTANT_np3
37:output_restart_check_INSTANT_np4
38:output_restart_check_AVERAGE_np1
39:output_restart_check_AVERAGE_np2
40:output_restart_check_AVERAGE_np3
41:output_restart_check_AVERAGE_np4
368:mam4_wetscav_standalone_baseline_cmp
458:model_initial_np4
459:model_restart_np4
460:restarted_vs_monolithic_check_np4
504:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp
  SCREAM V1 TESTING FAILED!
Waiting for tests to finish
FAIL ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240730_181332_ov2k5e
FAIL ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240730_181332_ov2k5e
FAIL ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240730_181332_ov2k5e
FAIL ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240730_181332_ov2k5e
PASS ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240730_181332_ov2k5e
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240730_181332_ov2k5e
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci.C.20240730_181332_ov2k5e
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240730_181332_ov2k5e
DIFF SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav.C.20240730_181332_ov2k5e
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240730_181332_ov2k5e
test-scheduler took 981.496740102768 seconds
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 45439 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh


We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs


squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel


[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins17186527877262667677.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: [email protected]
Finished: FAILURE
SCREAM_PullRequest_Autotester_Weaver # 5968 FAILED (click to see last 100 lines of console output) 

9:output_restart_np1
10:output_restart_check_INSTANT_np1
11:output_restart_check_AVERAGE_np1
109:mam4_wetscav_standalone_baseline_cmp
133:model_initial
134:model_restart
135:restarted_vs_monolithic_check_np1
146:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp

Error(s) occurred during test phase
OVERALL STATUS: FAIL
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5968/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5968/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5968/scream/components/eamxx
weaver failed'


    

  • SA_FAILURES_DETAILS+='Build type full_debug failed at testing time. Here'''s a list of failed tests:
9:output_restart_np1
10:output_restart_check_INSTANT_np1
11:output_restart_check_AVERAGE_np1
110:mam4_wetscav_standalone_baseline_cmp
134:model_initial
135:model_restart
136:restarted_vs_monolithic_check_np1
147:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp
    • 



Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:
9:output_restart_np1
10:output_restart_check_INSTANT_np1
11:output_restart_check_AVERAGE_np1
95:mam4_wetscav_standalone_baseline_cmp


Build type release failed at testing time. Here'''s a list of failed tests:
9:output_restart_np1
10:output_restart_check_INSTANT_np1
11:output_restart_check_AVERAGE_np1
109:mam4_wetscav_standalone_baseline_cmp
133:model_initial
134:model_restart
135:restarted_vs_monolithic_check_np1
146:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp


Error(s) occurred during test phase
OVERALL STATUS: FAIL
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5968/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5968/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5968/scream/components/eamxx
weaver failed'


    

  • [[ 1 == 0 ]]
    • 

  • [[ weaver == \m\a\p\p\y ]]
    • 

  • set +x
######################################################
FAILS DETECTED:
SCREAM STANDALONE TESTING FAILED!
Build type full_debug failed at testing time. Here's a list of failed tests:
9:output_restart_np1
10:output_restart_check_INSTANT_np1
11:output_restart_check_AVERAGE_np1
110:mam4_wetscav_standalone_baseline_cmp
134:model_initial
135:model_restart
136:restarted_vs_monolithic_check_np1
147:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp
    • 



Build type full_sp_debug failed at testing time. Here's a list of failed tests:
9:output_restart_np1
10:output_restart_check_INSTANT_np1
11:output_restart_check_AVERAGE_np1
95:mam4_wetscav_standalone_baseline_cmp


Build type release failed at testing time. Here's a list of failed tests:
9:output_restart_np1
10:output_restart_check_INSTANT_np1
11:output_restart_check_AVERAGE_np1
109:mam4_wetscav_standalone_baseline_cmp
133:model_initial
134:model_restart
135:restarted_vs_monolithic_check_np1
146:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp


Error(s) occurred during test phase
OVERALL STATUS: FAIL
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5968/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5968/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5968/scream/components/eamxx
weaver failed
######################################################
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le


cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins13051378025086807327.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: [email protected]
Finished: FAILURE

E3SM-Bot avatar Jul 31 '24 00:07 E3SM-Bot

Also, consider adding more verbose comments if you'd like, though having them provenanced here is potentially enough 😸

mahf708 avatar Aug 01 '24 19:08 mahf708

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

E3SM-Bot avatar Aug 01 '24 19:08 E3SM-Bot

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5744
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;I/O;AT: RETEST;bugfix
PULLREQUESTNUM 2929
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 0bdead86c0f3a3b240d92efd3076a62cdc67dd65
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d508361a8e53f672662912e6fa02e8074703029
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5973
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;I/O;AT: RETEST;bugfix
PULLREQUESTNUM 2929
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 0bdead86c0f3a3b240d92efd3076a62cdc67dd65
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d508361a8e53f672662912e6fa02e8074703029
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: bartgol/eamxx/hist-restart-fix
  • SHA: 0bdead86c0f3a3b240d92efd3076a62cdc67dd65
  • Mode: TEST_REPO

Pull Request Author: bartgol

E3SM-Bot avatar Aug 01 '24 19:08 E3SM-Bot

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5744
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;I/O;AT: RETEST;bugfix
PULLREQUESTNUM 2929
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 0bdead86c0f3a3b240d92efd3076a62cdc67dd65
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d508361a8e53f672662912e6fa02e8074703029
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5973
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;I/O;AT: RETEST;bugfix
PULLREQUESTNUM 2929
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 0bdead86c0f3a3b240d92efd3076a62cdc67dd65
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d508361a8e53f672662912e6fa02e8074703029
TEST_REPO_ALIAS SCREAM
SCREAM_PullRequest_Autotester_Mappy # 5744 FAILED (click to see last 100 lines of console output) 

PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240801_141433_qncrtl
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci.C.20240801_141433_qncrtl
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240801_141433_qncrtl
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav.C.20240801_141433_qncrtl
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240801_141433_qncrtl
test-scheduler took 974.6501774787903 seconds'
+ V1_FAILURES_DETAILS+='Waiting for tests to finish
FAIL ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240801_141433_qncrtl
FAIL ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240801_141433_qncrtl
FAIL ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240801_141433_qncrtl
FAIL ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240801_141433_qncrtl
PASS ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240801_141433_qncrtl
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240801_141433_qncrtl
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci.C.20240801_141433_qncrtl
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240801_141433_qncrtl
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav.C.20240801_141433_qncrtl
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240801_141433_qncrtl
test-scheduler took 974.6501774787903 seconds'
+ set +x
######################################################
FAILS DETECTED:
  SCREAM STANDALONE TESTING FAILED!
Build type full_debug failed at testing time. Here's a list of failed tests:
30:output_restart_np1
31:output_restart_np2
32:output_restart_np3
33:output_restart_np4
34:output_restart_check_INSTANT_np1
35:output_restart_check_INSTANT_np2
36:output_restart_check_INSTANT_np3
37:output_restart_check_INSTANT_np4
38:output_restart_check_AVERAGE_np1
39:output_restart_check_AVERAGE_np2
40:output_restart_check_AVERAGE_np3
41:output_restart_check_AVERAGE_np4
458:model_initial_np4
459:model_restart_np4
460:restarted_vs_monolithic_check_np4
  SCREAM V1 TESTING FAILED!
Waiting for tests to finish
FAIL ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240801_141433_qncrtl
FAIL ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240801_141433_qncrtl
FAIL ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240801_141433_qncrtl
FAIL ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240801_141433_qncrtl
PASS ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240801_141433_qncrtl
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240801_141433_qncrtl
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci.C.20240801_141433_qncrtl
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240801_141433_qncrtl
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav.C.20240801_141433_qncrtl
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240801_141433_qncrtl
test-scheduler took 974.6501774787903 seconds
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 125416 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh


We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs


squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel


[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins1951531489929453031.sh
POST BUILD TASK : FAILURE
END OF POST BUILD TASK : 0
Sending e-mails to: [email protected]
Finished: FAILURE
SCREAM_PullRequest_Autotester_Weaver # 5973 FAILED (click to see last 100 lines of console output) 

9:output_restart_np1
10:output_restart_check_INSTANT_np1
11:output_restart_check_AVERAGE_np1
109:mam4_wetscav_standalone_baseline_cmp
133:model_initial
134:model_restart
135:restarted_vs_monolithic_check_np1
146:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp

Error(s) occurred during test phase
OVERALL STATUS: FAIL
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5973/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5973/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5973/scream/components/eamxx
weaver failed'


    

  • SA_FAILURES_DETAILS+='Build type full_debug failed at testing time. Here'''s a list of failed tests:
9:output_restart_np1
10:output_restart_check_INSTANT_np1
11:output_restart_check_AVERAGE_np1
110:mam4_wetscav_standalone_baseline_cmp
134:model_initial
135:model_restart
136:restarted_vs_monolithic_check_np1
147:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp
    • 



Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:
9:output_restart_np1
10:output_restart_check_INSTANT_np1
11:output_restart_check_AVERAGE_np1
95:mam4_wetscav_standalone_baseline_cmp


Build type release failed at testing time. Here'''s a list of failed tests:
9:output_restart_np1
10:output_restart_check_INSTANT_np1
11:output_restart_check_AVERAGE_np1
109:mam4_wetscav_standalone_baseline_cmp
133:model_initial
134:model_restart
135:restarted_vs_monolithic_check_np1
146:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp


Error(s) occurred during test phase
OVERALL STATUS: FAIL
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5973/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5973/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5973/scream/components/eamxx
weaver failed'


    

  • [[ 1 == 0 ]]
    • 

  • [[ weaver == \m\a\p\p\y ]]
    • 

  • set +x
######################################################
FAILS DETECTED:
SCREAM STANDALONE TESTING FAILED!
Build type full_debug failed at testing time. Here's a list of failed tests:
9:output_restart_np1
10:output_restart_check_INSTANT_np1
11:output_restart_check_AVERAGE_np1
110:mam4_wetscav_standalone_baseline_cmp
134:model_initial
135:model_restart
136:restarted_vs_monolithic_check_np1
147:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp
    • 



Build type full_sp_debug failed at testing time. Here's a list of failed tests:
9:output_restart_np1
10:output_restart_check_INSTANT_np1
11:output_restart_check_AVERAGE_np1
95:mam4_wetscav_standalone_baseline_cmp


Build type release failed at testing time. Here's a list of failed tests:
9:output_restart_np1
10:output_restart_check_INSTANT_np1
11:output_restart_check_AVERAGE_np1
109:mam4_wetscav_standalone_baseline_cmp
133:model_initial
134:model_restart
135:restarted_vs_monolithic_check_np1
146:homme_shoc_cld_spa_p3_rrtmgp_mam4_wetscav_baseline_cmp


Error(s) occurred during test phase
OVERALL STATUS: FAIL
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5973/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5973/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5973/scream/components/eamxx
weaver failed
######################################################
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le


cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins8584003411079783548.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: [email protected]
Finished: FAILURE

E3SM-Bot avatar Aug 01 '24 20:08 E3SM-Bot

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

E3SM-Bot avatar Aug 07 '24 02:08 E3SM-Bot

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5765
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;I/O;AT: RETEST;AT: AUTOMERGE;bugfix
PULLREQUESTNUM 2929
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 0bdead86c0f3a3b240d92efd3076a62cdc67dd65
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d508361a8e53f672662912e6fa02e8074703029
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5991
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;I/O;AT: RETEST;AT: AUTOMERGE;bugfix
PULLREQUESTNUM 2929
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 0bdead86c0f3a3b240d92efd3076a62cdc67dd65
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d508361a8e53f672662912e6fa02e8074703029
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: bartgol/eamxx/hist-restart-fix
  • SHA: 0bdead86c0f3a3b240d92efd3076a62cdc67dd65
  • Mode: TEST_REPO

Pull Request Author: bartgol

E3SM-Bot avatar Aug 07 '24 11:08 E3SM-Bot

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5765
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;I/O;AT: RETEST;AT: AUTOMERGE;bugfix
PULLREQUESTNUM 2929
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 0bdead86c0f3a3b240d92efd3076a62cdc67dd65
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d508361a8e53f672662912e6fa02e8074703029
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5991
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;I/O;AT: RETEST;AT: AUTOMERGE;bugfix
PULLREQUESTNUM 2929
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 0bdead86c0f3a3b240d92efd3076a62cdc67dd65
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d508361a8e53f672662912e6fa02e8074703029
TEST_REPO_ALIAS SCREAM
SCREAM_PullRequest_Autotester_Mappy # 5765 FAILED (click to see last 100 lines of console output) 

FAIL ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240807_060452_nqfah2
FAIL ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240807_060452_nqfah2
FAIL ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240807_060452_nqfah2
FAIL ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240807_060452_nqfah2
PASS ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-arm97 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-arm97.C.20240807_060452_nqfah2
PASS ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-comble RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-comble.C.20240807_060452_nqfah2
PASS ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-dycomsrf01 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-dycomsrf01.C.20240807_060452_nqfah2
PASS ERS_P16_Ln22.ne30pg2_ne30pg2.FRCE-SCREAMv1-DP.mappy_gnu RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30pg2_ne30pg2.FRCE-SCREAMv1-DP.mappy_gnu.C.20240807_060452_nqfah2
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240807_060452_nqfah2
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci.C.20240807_060452_nqfah2
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240807_060452_nqfah2
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav.C.20240807_060452_nqfah2
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240807_060452_nqfah2
test-scheduler took 1206.1205797195435 seconds'
+ set +x
######################################################
FAILS DETECTED:
  SCREAM STANDALONE TESTING FAILED!
Build type full_debug failed at testing time. Here's a list of failed tests:
30:output_restart_np1
31:output_restart_np2
32:output_restart_np3
33:output_restart_np4
34:output_restart_check_INSTANT_np1
35:output_restart_check_INSTANT_np2
36:output_restart_check_INSTANT_np3
37:output_restart_check_INSTANT_np4
38:output_restart_check_AVERAGE_np1
39:output_restart_check_AVERAGE_np2
40:output_restart_check_AVERAGE_np3
41:output_restart_check_AVERAGE_np4
458:model_initial_np4
459:model_restart_np4
460:restarted_vs_monolithic_check_np4
  SCREAM V1 TESTING FAILED!
Waiting for tests to finish
FAIL ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240807_060452_nqfah2
FAIL ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240807_060452_nqfah2
FAIL ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240807_060452_nqfah2
FAIL ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (phase RUN)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240807_060452_nqfah2
PASS ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-arm97 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-arm97.C.20240807_060452_nqfah2
PASS ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-comble RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-comble.C.20240807_060452_nqfah2
PASS ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-dycomsrf01 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30pg2_ne30pg2.FIOP-SCREAMv1-DP.mappy_gnu.scream-dpxx-dycomsrf01.C.20240807_060452_nqfah2
PASS ERS_P16_Ln22.ne30pg2_ne30pg2.FRCE-SCREAMv1-DP.mappy_gnu RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30pg2_ne30pg2.FRCE-SCREAMv1-DP.mappy_gnu.C.20240807_060452_nqfah2
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240807_060452_nqfah2
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-aci.C.20240807_060452_nqfah2
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240807_060452_nqfah2
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-wetscav.C.20240807_060452_nqfah2
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240807_060452_nqfah2
test-scheduler took 1206.1205797195435 seconds
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 14178 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh


We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs


squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel


[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins13620547854317546321.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: [email protected]
Finished: FAILURE
SCREAM_PullRequest_Autotester_Weaver # 5991 FAILED (click to see last 100 lines of console output) 

10:output_restart_check_INSTANT_np1
11:output_restart_check_AVERAGE_np1
134:model_initial
135:model_restart
136:restarted_vs_monolithic_check_np1

Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:
9:output_restart_np1
10:output_restart_check_INSTANT_np1
11:output_restart_check_AVERAGE_np1


Build type release failed at testing time. Here'''s a list of failed tests:
9:output_restart_np1
10:output_restart_check_INSTANT_np1
11:output_restart_check_AVERAGE_np1
133:model_initial
134:model_restart
135:restarted_vs_monolithic_check_np1


Error(s) occurred during test phase
OVERALL STATUS: FAIL
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5991/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5991/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5991/scream/components/eamxx
weaver failed'


    

  • SA_FAILURES_DETAILS+='Build type full_debug failed at testing time. Here'''s a list of failed tests:
9:output_restart_np1
10:output_restart_check_INSTANT_np1
11:output_restart_check_AVERAGE_np1
134:model_initial
135:model_restart
136:restarted_vs_monolithic_check_np1
    • 



Build type full_sp_debug failed at testing time. Here'''s a list of failed tests:
9:output_restart_np1
10:output_restart_check_INSTANT_np1
11:output_restart_check_AVERAGE_np1


Build type release failed at testing time. Here'''s a list of failed tests:
9:output_restart_np1
10:output_restart_check_INSTANT_np1
11:output_restart_check_AVERAGE_np1
133:model_initial
134:model_restart
135:restarted_vs_monolithic_check_np1


Error(s) occurred during test phase
OVERALL STATUS: FAIL
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5991/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5991/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5991/scream/components/eamxx
weaver failed'


    

  • [[ 1 == 0 ]]
    • 

  • [[ weaver == \m\a\p\p\y ]]
    • 

  • set +x
######################################################
FAILS DETECTED:
SCREAM STANDALONE TESTING FAILED!
Build type full_debug failed at testing time. Here's a list of failed tests:
9:output_restart_np1
10:output_restart_check_INSTANT_np1
11:output_restart_check_AVERAGE_np1
134:model_initial
135:model_restart
136:restarted_vs_monolithic_check_np1
    • 



Build type full_sp_debug failed at testing time. Here's a list of failed tests:
9:output_restart_np1
10:output_restart_check_INSTANT_np1
11:output_restart_check_AVERAGE_np1


Build type release failed at testing time. Here's a list of failed tests:
9:output_restart_np1
10:output_restart_check_INSTANT_np1
11:output_restart_check_AVERAGE_np1
133:model_initial
134:model_restart
135:restarted_vs_monolithic_check_np1


Error(s) occurred during test phase
OVERALL STATUS: FAIL
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5991/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5991/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5991/scream/components/eamxx
weaver failed
######################################################
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le


cd $WORKSPACE/${BUILD_ID}/


./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins18325858727751452450.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: [email protected]
Finished: FAILURE

E3SM-Bot avatar Aug 07 '24 12:08 E3SM-Bot

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5766
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;I/O;AT: AUTOMERGE;bugfix
PULLREQUESTNUM 2929
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA cd19cdfe58780516c55f7776ff9f4c61f379a719
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d508361a8e53f672662912e6fa02e8074703029
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5992
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;I/O;AT: AUTOMERGE;bugfix
PULLREQUESTNUM 2929
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA cd19cdfe58780516c55f7776ff9f4c61f379a719
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d508361a8e53f672662912e6fa02e8074703029
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: bartgol/eamxx/hist-restart-fix
  • SHA: cd19cdfe58780516c55f7776ff9f4c61f379a719
  • Mode: TEST_REPO

Pull Request Author: bartgol

E3SM-Bot avatar Aug 07 '24 16:08 E3SM-Bot

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 5766
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;I/O;AT: AUTOMERGE;bugfix
PULLREQUESTNUM 2929
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA cd19cdfe58780516c55f7776ff9f4c61f379a719
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d508361a8e53f672662912e6fa02e8074703029
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 5992
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS BFB;I/O;AT: AUTOMERGE;bugfix
PULLREQUESTNUM 2929
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA cd19cdfe58780516c55f7776ff9f4c61f379a719
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 1d508361a8e53f672662912e6fa02e8074703029
TEST_REPO_ALIAS SCREAM

E3SM-Bot avatar Aug 07 '24 17:08 E3SM-Bot

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging THE LAST COMMIT TO THIS PULL REQUEST HAS NOT BEEN REVIEWED YET!

E3SM-Bot avatar Aug 07 '24 17:08 E3SM-Bot

All Jobs Finished; status = PASSED, target_sha=05a500edf5d5b9b490ca61c6c516098879e5f7a1, However Inspection must be performed before merge can occur...

E3SM-Bot avatar Aug 07 '24 17:08 E3SM-Bot

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging THE LAST COMMIT TO THIS PULL REQUEST HAS BEEN REVIEWED, BUT NOT ACCEPTED OR REQUIRES CHANGES!

E3SM-Bot avatar Aug 07 '24 17:08 E3SM-Bot

All Jobs Finished; status = PASSED, target_sha=05a500edf5d5b9b490ca61c6c516098879e5f7a1, However Inspection must be performed before merge can occur...

E3SM-Bot avatar Aug 07 '24 17:08 E3SM-Bot