scream Enable Subfield Slices Spanning a Range of indices (Issue #1312)

This PR adds functionality that allows a user to create a "multi-slice" subfield and gain access to the underlying view. That is, a multi-slice subfield is in contrast to the previous capability that sliced a single entry of a chosen dimension--e.g., view(:, 7, :) to use notation common to fortran/matlab, and expands this capability to allow for slicing a chosen dimension across a range of values--e.g., view(:, 2:7, :).

Summary of changes and capabilities:

Added overloads for Field::subfield() that takes the extra argument(s) index_beg, index_end to indicate the bounds of the slicing range.
Added Field::get_multi_sliced_view() that provides the view underlying the subfield as a multi-dimensional view with the same number of dimensions as the parent's associated view (as opposed to a single-sliced view that is (N - 1)-dimensional.
- Note: as implemented, this is only returned as a LayoutStride view, rather than LayoutRight.

I do need to add a few more tests and will push those changes soon, but I wanted to get this out there as a draft. Any feedback about my approach would be much-appreciated, since this is my first time doing a deep-dive into this code.

Closes #1312

Apr 04 '24 23:04 mjs271

Converting from draft to full PR after adding tests for f.subfield() and sf.get_multi_sliced_view(). Also cleaned up ekat::subview() , added tests over there, and brought the ekat submodule up to date.

Note: the ekat branch for the PR is over on my fork, since I don't yet have push access to E3SM-Project/EKAT.

Apr 10 '24 07:04 mjs271

Status Flag 'Pull Request AutoTester' - GitHub reports Mergeable status = False

Apr 10 '24 07:04 E3SM-Autotester

Note: the ekat branch for the PR is over on my fork, since I don't yet have push access to E3SM-Project/EKAT

@mjs271 , does this mean we will need a separate EKAT PR to accompany this one?

Apr 10 '24 13:04 AaronDonahue

Thanks. Quick question: For future reference, how hard would it be to generalize this to take noncontiguous slices? Say by passing a sorted/unsorted map/list?

@AaronDonahue there's a PR in EKAT upstream, but probably a clean EKAT submodule update is in order? https://github.com/E3SM-Project/EKAT/pull/333

Apr 10 '24 13:04 mahf708

@AaronDonahue @mahf708 That's correct--I've got a PR open on ekat, but I'm not a member of that repo, so this PR has the submodule commit pointed at my personal fork. Happy to resolve that however seems best

And, @mahf708, in response to your question, I've given it a little thought, and I don't think it'd be an enormous undertaking, but I do think it would require a new approach. The reason being that, as far as I know, kokkos allows slicing only with scalars, {std,Kokkos}::pair<>(), and Kokkos::ALL. As I was working on this code, my thought for the noncontiguous slicing approach was that it'd require taking individual contiguous slices and then concatenating. However, there's a nonzero chance there's another clever approach I'm not thinking of

It also occurred to me that it could be nice to be able to slice multiple dimensions at once, but I'm not sure whether there's a ready use case and also whether that might be approaching too much complexity

Apr 10 '24 16:04 mjs271

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5173
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	16d63ec1f1e2a1d356d2faf4d8bdd8cf3fbbe560
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5521
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	16d63ec1f1e2a1d356d2faf4d8bdd8cf3fbbe560
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: mjs271

Apr 10 '24 22:04 E3SM-Autotester

Thanks @mjs271!

As I was working on this code, my thought for the noncontiguous slicing approach was that it'd require taking individual contiguous slices and then concatenating.

Yep, that's what I was thinking too. We can circle back to it later when there's more of a need along with multi-dim slicing 😸

Apr 10 '24 22:04 mahf708

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5173
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	16d63ec1f1e2a1d356d2faf4d8bdd8cf3fbbe560
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5521
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	16d63ec1f1e2a1d356d2faf4d8bdd8cf3fbbe560
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5173 FAILED (click to see last 100 lines of console output)


Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive externals/scorpio_classic # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive components/eam/src/physics/cosp2/external # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive cime # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive externals/ekat # timeout=10
hudson.plugins.git.GitException: Command "git submodule update --init --recursive externals/ekat" returned status code 1:
stdout: Cloning into 'externals/ekat'...
stderr: Warning: Permanently added the ECDSA host key for IP address '140.82.114.3' to the list of known hosts.
fatal: reference is not a tree: 856f699d531faddd5a3626d62cc9992c0a8efdd0
Unable to checkout '856f699d531faddd5a3626d62cc9992c0a8efdd0' in submodule path 'externals/ekat'
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2846)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2185)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.lambda$execute$0(CliGitAPIImpl.java:1573)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:327)
at java.base/java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:184)
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.submitRemainingCommand(GitCommandsExecutor.java:77)
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:70)

Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to mappy
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1787)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
at hudson.remoting.Channel.call(Channel.java:1003)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:153)
at jdk.internal.reflect.GeneratedMethodAccessor232.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:138)
at jdk.proxy31/jdk.proxy31.$Proxy101.execute(Unknown Source)
at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:196)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1390)
at hudson.scm.SCM.checkout(SCM.java:540)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1248)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)
at hudson.model.Run.execute(Run.java:1895)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:101)
at hudson.model.Executor.run(Executor.java:442)
Caused: hudson.plugins.git.GitException
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.checkResult(GitCommandsExecutor.java:89)
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:69)
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:47)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.execute(CliGitAPIImpl.java:1576)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:170)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:377)
at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused: java.io.IOException: Could not perform submodule update
at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:201)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1390)
at hudson.scm.SCM.checkout(SCM.java:540)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1248)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)
at hudson.model.Run.execute(Run.java:1895)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:101)
at hudson.model.Executor.run(Executor.java:442)
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 73338 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins13237062574134617540.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: [email protected]
Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5521 PASSED (click to see last 100 lines of console output)


        Start 105: shoc_p3_nudged
105/122 Test #105: shoc_p3_nudged ..........................................   Passed    1.77 sec
        Start 106: shoc_p3_nudged_remapped
106/122 Test #106: shoc_p3_nudged_remapped .................................   Passed    1.96 sec
        Start 107: shoc_p3_nudging_glob_novert
107/122 Test #107: shoc_p3_nudging_glob_novert .............................   Passed    1.84 sec
        Start 108: homme_shoc_cld_p3_rrtmgp_np1
108/122 Test #108: homme_shoc_cld_p3_rrtmgp_np1 ............................   Passed    4.27 sec
        Start 109: homme_shoc_cld_p3_rrtmgp_baseline_cmp
109/122 Test #109: homme_shoc_cld_p3_rrtmgp_baseline_cmp ...................   Passed    0.15 sec
        Start 110: homme_shoc_cld_p3_rrtmgp_pg2_np1
110/122 Test #110: homme_shoc_cld_p3_rrtmgp_pg2_np1 ........................   Passed    4.15 sec
        Start 111: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp
111/122 Test #111: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp ...............   Passed    0.13 sec
        Start 112: model_baseline
112/122 Test #112: model_baseline ..........................................   Passed    5.23 sec
        Start 113: model_initial
113/122 Test #113: model_initial ...........................................   Passed    5.49 sec
        Start 114: model_restart
114/122 Test #114: model_restart ...........................................   Passed    4.82 sec
        Start 115: restarted_vs_monolithic_check_np1
115/122 Test #115: restarted_vs_monolithic_check_np1 .......................   Passed    0.08 sec
        Start 116: homme_shoc_cld_spa_p3_rrtmgp_np1
116/122 Test #116: homme_shoc_cld_spa_p3_rrtmgp_np1 ........................   Passed    4.94 sec
        Start 117: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp
117/122 Test #117: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp ...............   Passed    0.23 sec
        Start 118: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
118/122 Test #118: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1 ..............   Passed    5.77 sec
        Start 119: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
119/122 Test #119: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1 ...   Passed    1.15 sec
        Start 120: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
120/122 Test #120: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp .....   Passed    1.17 sec
        Start 121: homme_shoc_cld_spa_p3_rrtmgp_dp_np1
121/122 Test #121: homme_shoc_cld_spa_p3_rrtmgp_dp_np1 .....................   Passed   17.83 sec
        Start 122: homme_shoc_cld_spa_p3_rrtmgp_dp_baseline_cmp
122/122 Test #122: homme_shoc_cld_spa_p3_rrtmgp_dp_baseline_cmp ............   Passed    0.23 sec
100% tests passed, 0 tests failed out of 122
Label Time Summary:
baseline_cmp        =  91.85 secproc (14 tests)
baseline_gen        = 154.54 secproc (16 tests)
bfbhash             =   0.88 secproc (1 test)
check               =   0.85 secproc (1 test)
cld                 =   8.72 secproc (3 tests)
cld_fraction        =   1.84 secproc (1 test)
cxx baseline_cmp    =  10.85 secproc (2 tests)
diagnostics         =  30.72 secproc (18 tests)
driver              =  18.06 secproc (7 tests)
dynamics            =   5.57 secproc (3 tests)
fail                =  53.03 secproc (4 tests)
io                  =  37.48 secproc (12 tests)
nudging             =   7.09 secproc (2 tests)
p3                  =  98.73 secproc (7 tests)
p3_sk               = 107.63 secproc (2 tests)
physics             = 221.90 secproc (18 tests)
remap               =   3.19 secproc (1 test)
rrtmgp              =  16.87 secproc (8 tests)
shoc                =  12.27 secproc (7 tests)
spa                 =   9.17 secproc (4 tests)
surface_coupling    =   1.89 sec*proc (1 test)
Total Test time (real) = 779.67 sec
Testing '''16d63ec1f1e2a1d356d2faf4d8bdd8cf3fbbe560''' for test '''release'''
RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/ctest-build/release
Testing '''16d63ec1f1e2a1d356d2faf4d8bdd8cf3fbbe560''' for test '''full_sp_debug'''
RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug" '''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/ctest-build/full_sp_debug
Testing '''16d63ec1f1e2a1d356d2faf4d8bdd8cf3fbbe560''' for test '''full_debug'''
RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/ctest-build/full_debug
OVERALL STATUS: PASS
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx
Completed analysis on weaver'

[[ 0 != 0 ]]
[[ 1 == 0 ]]
[[ weaver == \m\a\p\p\y ]]
set +x
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins12955974405537412289.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: [email protected]
Finished: SUCCESS

Apr 10 '24 23:04 E3SM-Autotester

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5179
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	52500ae71e84bf9119286e4abc82d398ecd66f96
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5523
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	52500ae71e84bf9119286e4abc82d398ecd66f96
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: mjs271

Apr 11 '24 12:04 E3SM-Autotester

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5179
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	52500ae71e84bf9119286e4abc82d398ecd66f96
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5523
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	52500ae71e84bf9119286e4abc82d398ecd66f96
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5179 FAILED (click to see last 100 lines of console output)


Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive externals/scorpio_classic # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive components/eam/src/physics/cosp2/external # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive cime # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive externals/ekat # timeout=10
hudson.plugins.git.GitException: Command "git submodule update --init --recursive externals/ekat" returned status code 1:
stdout: Cloning into 'externals/ekat'...
stderr: Warning: Permanently added the ECDSA host key for IP address '140.82.113.3' to the list of known hosts.
fatal: reference is not a tree: 1862414b96d023b53a36c5d5dcc968b248b60d0d
Unable to checkout '1862414b96d023b53a36c5d5dcc968b248b60d0d' in submodule path 'externals/ekat'
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2846)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2185)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.lambda$execute$0(CliGitAPIImpl.java:1573)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:327)
at java.base/java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:184)
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.submitRemainingCommand(GitCommandsExecutor.java:77)
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:70)

Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to mappy
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1787)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
at hudson.remoting.Channel.call(Channel.java:1003)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:153)
at jdk.internal.reflect.GeneratedMethodAccessor153.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:138)
at jdk.proxy29/jdk.proxy29.$Proxy158.execute(Unknown Source)
at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:196)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1390)
at hudson.scm.SCM.checkout(SCM.java:540)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1248)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)
at hudson.model.Run.execute(Run.java:1895)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:101)
at hudson.model.Executor.run(Executor.java:442)
Caused: hudson.plugins.git.GitException
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.checkResult(GitCommandsExecutor.java:89)
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:69)
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:47)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.execute(CliGitAPIImpl.java:1576)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:170)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:377)
at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused: java.io.IOException: Could not perform submodule update
at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:201)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1390)
at hudson.scm.SCM.checkout(SCM.java:540)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1248)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)
at hudson.model.Run.execute(Run.java:1895)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:101)
at hudson.model.Executor.run(Executor.java:442)
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 15198 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins5730545843742489020.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: [email protected]
Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5523 PASSED (click to see last 100 lines of console output)


        Start 112: shoc_p3_nudging_glob_novert
112/129 Test #112: shoc_p3_nudging_glob_novert .............................   Passed    1.98 sec
        Start 113: homme_shoc_cld_p3_rrtmgp_np1
113/129 Test #113: homme_shoc_cld_p3_rrtmgp_np1 ............................   Passed    6.01 sec
        Start 114: homme_shoc_cld_p3_rrtmgp_baseline_cmp
114/129 Test #114: homme_shoc_cld_p3_rrtmgp_baseline_cmp ...................   Passed    0.20 sec
        Start 115: homme_shoc_cld_p3_rrtmgp_pg2_np1
115/129 Test #115: homme_shoc_cld_p3_rrtmgp_pg2_np1 ........................   Passed    7.40 sec
        Start 116: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp
116/129 Test #116: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp ...............   Passed    0.18 sec
        Start 117: model_baseline
117/129 Test #117: model_baseline ..........................................   Passed    7.01 sec
        Start 118: model_initial
118/129 Test #118: model_initial ...........................................   Passed    4.22 sec
        Start 119: model_restart
119/129 Test #119: model_restart ...........................................   Passed    4.81 sec
        Start 120: restarted_vs_monolithic_check_np1
120/129 Test #120: restarted_vs_monolithic_check_np1 .......................   Passed    0.08 sec
        Start 121: homme_shoc_cld_spa_p3_rrtmgp_np1
121/129 Test #121: homme_shoc_cld_spa_p3_rrtmgp_np1 ........................   Passed    7.94 sec
        Start 122: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp
122/129 Test #122: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp ...............   Passed    0.21 sec
        Start 123: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
123/129 Test #123: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1 ..............   Passed    6.62 sec
        Start 124: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
124/129 Test #124: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1 ...   Passed    1.16 sec
        Start 125: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
125/129 Test #125: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp .....   Passed    0.86 sec
        Start 126: homme_shoc_cld_spa_p3_rrtmgp_dp_np1
126/129 Test #126: homme_shoc_cld_spa_p3_rrtmgp_dp_np1 .....................   Passed   20.11 sec
        Start 127: homme_shoc_cld_spa_p3_rrtmgp_dp_baseline_cmp
127/129 Test #127: homme_shoc_cld_spa_p3_rrtmgp_dp_baseline_cmp ............   Passed    0.26 sec
        Start 128: homme_shoc_cld_p3_mam_optics_rrtmgp_np1
128/129 Test #128: homme_shoc_cld_p3_mam_optics_rrtmgp_np1 .................   Passed    9.61 sec
        Start 129: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp
129/129 Test #129: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp ........   Passed    0.19 sec
100% tests passed, 0 tests failed out of 129
Label Time Summary:
baseline_cmp        =  60.47 secproc (16 tests)
baseline_gen        = 145.91 secproc (18 tests)
bfbhash             =   0.86 secproc (1 test)
check               =   0.77 secproc (1 test)
cld                 =   9.60 secproc (3 tests)
cld_fraction        =   2.33 secproc (1 test)
cxx baseline_cmp    =   8.40 secproc (2 tests)
diagnostics         =  37.05 secproc (19 tests)
driver              =  24.66 secproc (8 tests)
dynamics            =   5.23 secproc (3 tests)
fail                =  40.14 secproc (4 tests)
io                  =  36.30 secproc (12 tests)
mam4_optics         =   5.70 secproc (1 test)
nudging             =   8.27 secproc (2 tests)
p3                  = 135.47 secproc (7 tests)
p3_sk               =  41.88 secproc (2 tests)
physics             = 198.59 secproc (19 tests)
remap               =   3.20 secproc (1 test)
rrtmgp              =  22.60 secproc (9 tests)
shoc                =  13.12 secproc (7 tests)
spa                 =   9.60 secproc (4 tests)
surface_coupling    =   2.28 secproc (1 test)
Total Test time (real) = 582.95 sec
Testing '''d9c0bc846ffc60d14b68586182183bfa9a3c359d''' for test '''full_sp_debug'''
RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug" '''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/ctest-build/full_sp_debug
Testing '''d9c0bc846ffc60d14b68586182183bfa9a3c359d''' for test '''release'''
RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/ctest-build/release
Testing '''d9c0bc846ffc60d14b68586182183bfa9a3c359d''' for test '''full_debug'''
RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/ctest-build/full_debug
OVERALL STATUS: PASS
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx
Completed analysis on weaver'

[[ 0 != 0 ]]
[[ 1 == 0 ]]
[[ weaver == \m\a\p\p\y ]]
set +x
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins3502710905521686397.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Finished: SUCCESS

Apr 11 '24 13:04 E3SM-Autotester

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5182
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	52500ae71e84bf9119286e4abc82d398ecd66f96
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5526
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	52500ae71e84bf9119286e4abc82d398ecd66f96
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: mjs271

Apr 11 '24 15:04 E3SM-Autotester

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5182
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	52500ae71e84bf9119286e4abc82d398ecd66f96
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5526
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	52500ae71e84bf9119286e4abc82d398ecd66f96
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5182 FAILED (click to see last 100 lines of console output)


Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive externals/scorpio_classic # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive components/eam/src/physics/cosp2/external # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive cime # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive externals/ekat # timeout=10
hudson.plugins.git.GitException: Command "git submodule update --init --recursive externals/ekat" returned status code 1:
stdout: Cloning into 'externals/ekat'...
stderr: Warning: Permanently added the ECDSA host key for IP address '140.82.113.4' to the list of known hosts.
fatal: reference is not a tree: 1862414b96d023b53a36c5d5dcc968b248b60d0d
Unable to checkout '1862414b96d023b53a36c5d5dcc968b248b60d0d' in submodule path 'externals/ekat'
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2846)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2185)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.lambda$execute$0(CliGitAPIImpl.java:1573)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:327)
at java.base/java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:184)
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.submitRemainingCommand(GitCommandsExecutor.java:77)
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:70)

Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to mappy
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1787)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
at hudson.remoting.Channel.call(Channel.java:1003)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:153)
at jdk.internal.reflect.GeneratedMethodAccessor153.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:138)
at jdk.proxy29/jdk.proxy29.$Proxy158.execute(Unknown Source)
at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:196)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1390)
at hudson.scm.SCM.checkout(SCM.java:540)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1248)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)
at hudson.model.Run.execute(Run.java:1895)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:101)
at hudson.model.Executor.run(Executor.java:442)
Caused: hudson.plugins.git.GitException
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.checkResult(GitCommandsExecutor.java:89)
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:69)
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:47)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.execute(CliGitAPIImpl.java:1576)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:170)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:377)
at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused: java.io.IOException: Could not perform submodule update
at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:201)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1390)
at hudson.scm.SCM.checkout(SCM.java:540)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1248)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)
at hudson.model.Run.execute(Run.java:1895)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:101)
at hudson.model.Executor.run(Executor.java:442)
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 123157 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins11852718006915602681.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: [email protected]
Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5526 PASSED (click to see last 100 lines of console output)


        Start 112: shoc_p3_nudging_glob_novert
112/129 Test #112: shoc_p3_nudging_glob_novert .............................   Passed    2.74 sec
        Start 113: homme_shoc_cld_p3_rrtmgp_np1
113/129 Test #113: homme_shoc_cld_p3_rrtmgp_np1 ............................   Passed    6.64 sec
        Start 114: homme_shoc_cld_p3_rrtmgp_baseline_cmp
114/129 Test #114: homme_shoc_cld_p3_rrtmgp_baseline_cmp ...................   Passed    0.18 sec
        Start 115: homme_shoc_cld_p3_rrtmgp_pg2_np1
115/129 Test #115: homme_shoc_cld_p3_rrtmgp_pg2_np1 ........................   Passed    6.99 sec
        Start 116: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp
116/129 Test #116: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp ...............   Passed    0.11 sec
        Start 117: model_baseline
117/129 Test #117: model_baseline ..........................................   Passed    7.51 sec
        Start 118: model_initial
118/129 Test #118: model_initial ...........................................   Passed    4.23 sec
        Start 119: model_restart
119/129 Test #119: model_restart ...........................................   Passed    4.79 sec
        Start 120: restarted_vs_monolithic_check_np1
120/129 Test #120: restarted_vs_monolithic_check_np1 .......................   Passed    0.10 sec
        Start 121: homme_shoc_cld_spa_p3_rrtmgp_np1
121/129 Test #121: homme_shoc_cld_spa_p3_rrtmgp_np1 ........................   Passed    6.29 sec
        Start 122: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp
122/129 Test #122: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp ...............   Passed    0.16 sec
        Start 123: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
123/129 Test #123: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1 ..............   Passed    6.77 sec
        Start 124: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
124/129 Test #124: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1 ...   Passed    1.17 sec
        Start 125: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
125/129 Test #125: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp .....   Passed    1.01 sec
        Start 126: homme_shoc_cld_spa_p3_rrtmgp_dp_np1
126/129 Test #126: homme_shoc_cld_spa_p3_rrtmgp_dp_np1 .....................   Passed   20.06 sec
        Start 127: homme_shoc_cld_spa_p3_rrtmgp_dp_baseline_cmp
127/129 Test #127: homme_shoc_cld_spa_p3_rrtmgp_dp_baseline_cmp ............   Passed    0.27 sec
        Start 128: homme_shoc_cld_p3_mam_optics_rrtmgp_np1
128/129 Test #128: homme_shoc_cld_p3_mam_optics_rrtmgp_np1 .................   Passed    7.38 sec
        Start 129: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp
129/129 Test #129: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp ........   Passed    0.20 sec
100% tests passed, 0 tests failed out of 129
Label Time Summary:
baseline_cmp        =  59.37 secproc (16 tests)
baseline_gen        = 144.95 secproc (18 tests)
bfbhash             =   0.79 secproc (1 test)
check               =   0.72 secproc (1 test)
cld                 =  11.86 secproc (3 tests)
cld_fraction        =   2.14 secproc (1 test)
cxx baseline_cmp    =   9.28 secproc (2 tests)
diagnostics         =  38.14 secproc (19 tests)
driver              =  29.65 secproc (8 tests)
dynamics            =   5.15 secproc (3 tests)
fail                =  40.33 secproc (4 tests)
io                  =  42.27 secproc (12 tests)
mam4_optics         =   5.18 secproc (1 test)
nudging             =   9.62 secproc (2 tests)
p3                  = 131.79 secproc (7 tests)
p3_sk               =  97.02 secproc (2 tests)
physics             = 251.34 secproc (19 tests)
remap               =   4.59 secproc (1 test)
rrtmgp              =  25.72 secproc (9 tests)
shoc                =  15.75 secproc (7 tests)
spa                 =  10.61 secproc (4 tests)
surface_coupling    =   4.35 secproc (1 test)
Total Test time (real) = 647.87 sec
Testing '''50f995e8afe99f8bbef9242a1f3806095ed19f17''' for test '''release'''
RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/ctest-build/release
Testing '''50f995e8afe99f8bbef9242a1f3806095ed19f17''' for test '''full_sp_debug'''
RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug" '''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/ctest-build/full_sp_debug
Testing '''50f995e8afe99f8bbef9242a1f3806095ed19f17''' for test '''full_debug'''
RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/ctest-build/full_debug
OVERALL STATUS: PASS
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx
Completed analysis on weaver'

[[ 0 != 0 ]]
[[ 1 == 0 ]]
[[ weaver == \m\a\p\p\y ]]
set +x
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins15109751008381081617.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Finished: SUCCESS

Apr 11 '24 16:04 E3SM-Autotester

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5186
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	52500ae71e84bf9119286e4abc82d398ecd66f96
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5530
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	52500ae71e84bf9119286e4abc82d398ecd66f96
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: mjs271

Apr 11 '24 20:04 E3SM-Autotester

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5186
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	52500ae71e84bf9119286e4abc82d398ecd66f96
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5530
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	52500ae71e84bf9119286e4abc82d398ecd66f96
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5186 FAILED (click to see last 100 lines of console output)


Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive externals/scorpio_classic # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive components/eam/src/physics/cosp2/external # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive cime # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive externals/ekat # timeout=10
hudson.plugins.git.GitException: Command "git submodule update --init --recursive externals/ekat" returned status code 1:
stdout: Cloning into 'externals/ekat'...
stderr: Warning: Permanently added the ECDSA host key for IP address '140.82.114.3' to the list of known hosts.
fatal: reference is not a tree: 1862414b96d023b53a36c5d5dcc968b248b60d0d
Unable to checkout '1862414b96d023b53a36c5d5dcc968b248b60d0d' in submodule path 'externals/ekat'
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2846)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2185)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.lambda$execute$0(CliGitAPIImpl.java:1573)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:327)
at java.base/java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:184)
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.submitRemainingCommand(GitCommandsExecutor.java:77)
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:70)

Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to mappy
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1787)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
at hudson.remoting.Channel.call(Channel.java:1003)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:153)
at jdk.internal.reflect.GeneratedMethodAccessor153.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:138)
at jdk.proxy29/jdk.proxy29.$Proxy158.execute(Unknown Source)
at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:196)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1390)
at hudson.scm.SCM.checkout(SCM.java:540)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1248)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)
at hudson.model.Run.execute(Run.java:1895)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:101)
at hudson.model.Executor.run(Executor.java:442)
Caused: hudson.plugins.git.GitException
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.checkResult(GitCommandsExecutor.java:89)
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:69)
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:47)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.execute(CliGitAPIImpl.java:1576)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:170)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:377)
at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused: java.io.IOException: Could not perform submodule update
at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:201)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1390)
at hudson.scm.SCM.checkout(SCM.java:540)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1248)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)
at hudson.model.Run.execute(Run.java:1895)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
at hudson.model.ResourceController.execute(ResourceController.java:101)
at hudson.model.Executor.run(Executor.java:442)
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 119445 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins316686276118700470.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: [email protected]
Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5530 PASSED (click to see last 100 lines of console output)


        Start 112: shoc_p3_nudging_glob_novert
112/129 Test #112: shoc_p3_nudging_glob_novert .............................   Passed    1.90 sec
        Start 113: homme_shoc_cld_p3_rrtmgp_np1
113/129 Test #113: homme_shoc_cld_p3_rrtmgp_np1 ............................   Passed    7.58 sec
        Start 114: homme_shoc_cld_p3_rrtmgp_baseline_cmp
114/129 Test #114: homme_shoc_cld_p3_rrtmgp_baseline_cmp ...................   Passed    0.15 sec
        Start 115: homme_shoc_cld_p3_rrtmgp_pg2_np1
115/129 Test #115: homme_shoc_cld_p3_rrtmgp_pg2_np1 ........................   Passed    7.44 sec
        Start 116: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp
116/129 Test #116: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp ...............   Passed    0.13 sec
        Start 117: model_baseline
117/129 Test #117: model_baseline ..........................................   Passed    8.39 sec
        Start 118: model_initial
118/129 Test #118: model_initial ...........................................   Passed    4.24 sec
        Start 119: model_restart
119/129 Test #119: model_restart ...........................................   Passed    4.81 sec
        Start 120: restarted_vs_monolithic_check_np1
120/129 Test #120: restarted_vs_monolithic_check_np1 .......................   Passed    0.09 sec
        Start 121: homme_shoc_cld_spa_p3_rrtmgp_np1
121/129 Test #121: homme_shoc_cld_spa_p3_rrtmgp_np1 ........................   Passed    8.10 sec
        Start 122: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp
122/129 Test #122: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp ...............   Passed    0.22 sec
        Start 123: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
123/129 Test #123: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1 ..............   Passed    6.80 sec
        Start 124: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
124/129 Test #124: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1 ...   Passed    1.17 sec
        Start 125: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
125/129 Test #125: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp .....   Passed    0.87 sec
        Start 126: homme_shoc_cld_spa_p3_rrtmgp_dp_np1
126/129 Test #126: homme_shoc_cld_spa_p3_rrtmgp_dp_np1 .....................   Passed   20.02 sec
        Start 127: homme_shoc_cld_spa_p3_rrtmgp_dp_baseline_cmp
127/129 Test #127: homme_shoc_cld_spa_p3_rrtmgp_dp_baseline_cmp ............   Passed    0.21 sec
        Start 128: homme_shoc_cld_p3_mam_optics_rrtmgp_np1
128/129 Test #128: homme_shoc_cld_p3_mam_optics_rrtmgp_np1 .................   Passed   10.32 sec
        Start 129: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp
129/129 Test #129: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp ........   Passed    0.17 sec
100% tests passed, 0 tests failed out of 129
Label Time Summary:
baseline_cmp        =  86.20 secproc (16 tests)
baseline_gen        = 178.18 secproc (18 tests)
bfbhash             =   0.88 secproc (1 test)
check               =   0.80 secproc (1 test)
cld                 =  10.57 secproc (3 tests)
cld_fraction        =   2.29 secproc (1 test)
cxx baseline_cmp    =   7.13 secproc (2 tests)
diagnostics         =  46.00 secproc (19 tests)
driver              =  30.24 secproc (8 tests)
dynamics            =   5.08 secproc (3 tests)
fail                =  48.93 secproc (4 tests)
io                  =  42.13 secproc (12 tests)
mam4_optics         =   5.82 secproc (1 test)
nudging             =  10.17 secproc (2 tests)
p3                  = 142.07 secproc (7 tests)
p3_sk               = 123.55 secproc (2 tests)
physics             = 292.73 secproc (19 tests)
remap               =   5.99 secproc (1 test)
rrtmgp              =  28.01 secproc (9 tests)
shoc                =  14.65 secproc (7 tests)
spa                 =   9.60 secproc (4 tests)
surface_coupling    =   3.73 secproc (1 test)
Total Test time (real) = 711.15 sec
Testing '''a32dba4fdd314a7622254a344dcb94e1cb631ca7''' for test '''full_sp_debug'''
RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug" '''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/ctest-build/full_sp_debug
Testing '''a32dba4fdd314a7622254a344dcb94e1cb631ca7''' for test '''release'''
RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/ctest-build/release
Testing '''a32dba4fdd314a7622254a344dcb94e1cb631ca7''' for test '''full_debug'''
RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '''
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/ctest-build/full_debug
OVERALL STATUS: PASS
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx
Completed analysis on weaver'

[[ 0 != 0 ]]
[[ 1 == 0 ]]
[[ weaver == \m\a\p\p\y ]]
set +x
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins720872902790362728.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Finished: SUCCESS

Apr 11 '24 21:04 E3SM-Autotester

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5190
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	f5bfe88658d10fd922d247cfa9522f365eb26c0b
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5534
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	f5bfe88658d10fd922d247cfa9522f365eb26c0b
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: mjs271

Apr 11 '24 23:04 E3SM-Autotester

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5190
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	f5bfe88658d10fd922d247cfa9522f365eb26c0b
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5534
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	f5bfe88658d10fd922d247cfa9522f365eb26c0b
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5190 FAILED (click to see last 100 lines of console output)


    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240411_180348_9eaeke
DIFF ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240411_180348_9eaeke
DIFF ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240411_180348_9eaeke
DIFF PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240411_180348_9eaeke
DIFF SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240411_180348_9eaeke
DIFF SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240411_180348_9eaeke
test-scheduler took 796.1405007839203 seconds'
+ errors='Waiting for tests to finish
DIFF ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240411_180348_9eaeke
DIFF ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240411_180348_9eaeke
DIFF ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240411_180348_9eaeke
DIFF ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240411_180348_9eaeke
DIFF ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240411_180348_9eaeke
DIFF PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240411_180348_9eaeke
DIFF SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240411_180348_9eaeke
DIFF SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240411_180348_9eaeke
test-scheduler took 796.1405007839203 seconds'
+ V1_FAILURES_DETAILS+='Waiting for tests to finish
DIFF ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240411_180348_9eaeke
DIFF ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240411_180348_9eaeke
DIFF ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240411_180348_9eaeke
DIFF ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240411_180348_9eaeke
DIFF ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240411_180348_9eaeke
DIFF PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240411_180348_9eaeke
DIFF SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240411_180348_9eaeke
DIFF SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240411_180348_9eaeke
test-scheduler took 796.1405007839203 seconds'
+ set +x
######################################################
FAILS DETECTED:
  SCREAM STANDALONE TESTING FAILED!
Build type full_debug failed at build time. Here's the build log:
Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5190/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m mappy
RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5190/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m mappy
FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5190/scream/components/eamxx
mappy failed
  SCREAM V1 TESTING FAILED!
Waiting for tests to finish
DIFF ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240411_180348_9eaeke
DIFF ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240411_180348_9eaeke
DIFF ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240411_180348_9eaeke
DIFF ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240411_180348_9eaeke
DIFF ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240411_180348_9eaeke
DIFF PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240411_180348_9eaeke
DIFF SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240411_180348_9eaeke
DIFF SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240411_180348_9eaeke
test-scheduler took 796.1405007839203 seconds
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 20978 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins3388199214036993032.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: [email protected]
Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5534 FAILED (click to see last 100 lines of console output)


  534 |                         pg%g_lat, pg%g_dofs_per_rank, pg%g_dofs_offsets, MPIreal_t, &
      |                        1
......
  593 |             dofs_per_elem, g_elem_per_rank, g_elem_offsets, MPIinteger_t, par%comm, ierr)
      |            2            
Warning: Type mismatch between actual argument at (1) and actual argument at (2) (REAL(8)/INTEGER(4)).
[ 75%] Linking CXX executable io_diags
[ 75%] Built target property_checks
[ 75%] Building Fortran object src/dynamics/homme/tests/CMakeFiles/scream_theta-l_kokkos_4_72_10.dir/__/interface/homme_grid_mod.F90.o
[ 75%] Built target io_se_grid
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx/src/physics/p3/tests/p3_upwind_unit_tests.cpp: In function 'scream::p3::unit_test::UnitWrap::UnitTest<:device kokkos::cudaspace> >::TestUpwind::run_phys()::':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx/src/physics/p3/tests/p3_upwind_unit_tests.cpp:190:22: warning: 'r_min0' may be used uninitialized [-Wmaybe-uninitialized]
  190 |           if (r_min1  r_max0 + 10*eps) ++nerr;
      |              ~~~~~~~~^~~~~~~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx/src/physics/p3/tests/p3_upwind_unit_tests.cpp:150:23: note: 'r_max0' was declared here
  150 |           Scalar mass0, r_min0, r_max0;
      |                       ^~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx/src/physics/p3/tests/p3_upwind_unit_tests.cpp:191:1: warning: 'r_max1' may be used uninitialized [-Wmaybe-uninitialized]
  191 |           if (r_max1 > r_max0 + 10*eps) ++nerr;
      | ^ 
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx/src/physics/p3/tests/p3_upwind_unit_tests.cpp:180:23: note: 'r_max1' was declared here
  180 |           Scalar mass1, r_min1, r_max1;
      |                       ^~~~~~
[ 75%] Built target io_filled
nvcc_internal_extended_lambda_implementation: In static member function 'static void* __nv_hdl_wrapper_t::manager::do_copy(void*) [with Lambda = scream::p3::unit_test::UnitWrap::UnitTest<:device kokkos::cudaspace> >::TestUpwind::run_phys()::; Tag = __nv_dl_tag >::TestUpwind::run_phys, 2>; OpFuncR = void; OpFuncArgs = {const Kokkos::Impl::CudaTeamMember&, int&}; F1 = Kokkos::Array<:view>*, Kokkos::LayoutRight, Kokkos::Device<:cuda kokkos::cudaspace>, Kokkos::MemoryTraits >, 2, void>; F2 = Kokkos::View<:pack>*, Kokkos::LayoutRight, Kokkos::Device<:cuda kokkos::cudaspace>, Kokkos::MemoryTraits >; F3 = Kokkos::View<:pack>*, Kokkos::LayoutRight, Kokkos::Device<:cuda kokkos::cudaspace>, Kokkos::MemoryTraits >; F4 = int; F5 = const double; F6 = const int; F7 = const int; F8 = int; F9 = const Kokkos::View<:pack>*, Kokkos::LayoutRight, Kokkos::Device<:cuda kokkos::cudaspace>, Kokkos::MemoryTraits >; F10 = const Kokkos::View<:pack>*, Kokkos::LayoutRight, Kokkos::Device<:cuda kokkos::cudaspace>, Kokkos::MemoryTraits >; F11 = const Kokkos::View<:pack>*, Kokkos::LayoutRight, Kokkos::Device<:cuda kokkos::cudaspace>, Kokkos::MemoryTraits >; F12 = int; F13 = const double; F14 = Kokkos::Array<:view>*, Kokkos::LayoutRight, Kokkos::Device<:cuda kokkos::cudaspace>, Kokkos::MemoryTraits >, 2, void>; F15 = Kokkos::Array<:view>*, Kokkos::LayoutRight, Kokkos::Device<:cuda kokkos::cudaspace>, Kokkos::MemoryTraits >, 2, void>; F16 = Kokkos::Array<:view>*, Kokkos::LayoutRight, Kokkos::Device<:cuda kokkos::cudaspace>, Kokkos::MemoryTraits >, 2, void>; F17 = const Kokkos::View, Kokkos::MemoryTraits >]':
nvcc_internal_extended_lambda_implementation:379:19: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
[ 75%] Built target io_diags
[ 75%] Linking CXX executable output_restart
[ 75%] Linking CXX executable field_utils
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx/src/share/field/field_alloc_prop.hpp: In copy constructor 'scream::FieldAllocProp::FieldAllocProp(const scream::FieldAllocProp&)':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx/src/share/field/field_alloc_prop.hpp:105:1: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  105 |   FieldAllocProp (const FieldAllocProp&) = default;
      | ^ ~~~~~~~~~~~~
[ 75%] Built target output_restart
/projects/ppc64le-pwr9-rhel8/compilers/gcc/11.3.0/gcc/8.3.1/base/tchbki3/lib/gcc/powerpc64le-unknown-linux-gnu/11.3.0/../../../../include/c++/11.3.0/bits/stl_pair.h: In instantiation of 'constexpr std::pair::type>::__type, typename std::__strip_reference_wrapper::type>::__type> std::make_pair(_T1&&, _T2&&) [with _T1 = double; _T2 = double; typename std::__strip_reference_wrapper::type>::__type = double; typename std::decay<_tp2>::type = double; typename std::__strip_reference_wrapper::type>::__type = double; typename std::decay<_tp>::type = double]':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx/src/physics/p3/tests/p3_ice_sed_unit_tests.cpp:204:37:   required from 'static void scream::p3::unit_test::UnitWrap::UnitTest::TestIceSed::run_bfb_homogeneous_freezing() [with D = Kokkos::Device<:cuda kokkos::cudaspace>]'
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx/src/physics/p3/tests/p3_ice_sed_unit_tests.cpp:253:31:   required from 'static void scream::p3::unit_test::UnitWrap::UnitTest::TestIceSed::run_bfb() [with D = Kokkos::Device<:cuda kokkos::cudaspace>]'
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx/src/physics/p3/tests/p3_ice_sed_unit_tests.cpp:269:6:   required from here
/projects/ppc64le-pwr9-rhel8/compilers/gcc/11.3.0/gcc/8.3.1/base/tchbki3/lib/gcc/powerpc64le-unknown-linux-gnu/11.3.0/../../../../include/c++/11.3.0/bits/stl_pair.h:567:1: note: parameter passing for argument of type 'std::pair' when C++17 is enabled changed to match C++14 in GCC 10.1
  567 |     make_pair(_T1&& __x, _T2&& __y)
      | ^   ~~~~~
[ 75%] Built target field_utils
[ 76%] Linking CXX executable time_interpolation
[ 76%] Linking CXX executable atm_proc
[ 76%] Built target time_interpolation
[ 76%] Built target atm_proc
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/externals/ekat/extern/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Range.hpp: In member function 'void Kokkos::Impl::ParallelReduce, ReducerType, Kokkos::Cuda>::execute() [with FunctorType = __nv_hdl_wrapper_t >::TestP3CloudWaterAutoconversion::run_physics, 1>, void(const int&, int&)>; ReducerType = Kokkos::InvalidType; Traits = {Kokkos::Cuda}]':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/externals/ekat/extern/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Range.hpp:321:6: note: the layout of aggregates containing vectors with 2-byte alignment has changed in GCC 5
  321 |   inline void execute() {
      |      ^~~~~~~
[ 76%] Linking CXX static library libscream_rrtmgp.a
[ 76%] Built target scream_rrtmgp
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx/src/physics/p3/tests/p3_ice_nucleation_unit_tests.cpp: In static member function 'static void* __nv_hdl_wrapper_t::manager::do_copy(void*) [with Lambda = scream::p3::unit_test::UnitWrap::UnitTest<:device kokkos::cudaspace> >::TestIceNucleation::run_ice_nucleation_bfb()::; Tag = __nv_dl_tag >::TestIceNucleation::run_ice_nucleation_bfb, 1>; OpFuncR = void; OpFuncArgs = {const int&}; F1 = Kokkos::View<:p3::icenucleationdata kokkos::layoutright kokkos::device kokkos::cudaspace>, Kokkos::MemoryTraits >; F2 = bool; F3 = bool]':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx/src/physics/p3/tests/p3_ice_nucleation_unit_tests.cpp:69:348: note: the layout of aggregates containing vectors with 2-byte alignment has changed in GCC 5
   69 |         Kokkos::parallel_for(num_test_itrs, KOKKOS_LAMBDA(const Int& i) {
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/externals/ekat/extern/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Team.hpp: In member function 'void Kokkos::Impl::ParallelReduce, ReducerType, Kokkos::Cuda>::execute() [with FunctorType = __nv_hdl_wrapper_t >::TestP3SubgridVarianceScaling::run_property_tests, 1>, void(const Kokkos::Impl::CudaTeamMember&, int&)>; ReducerType = Kokkos::InvalidType; Properties = {Kokkos::Cuda}]':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/externals/ekat/extern/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Team.hpp:839:6: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  839 |   inline void execute() {
      |      ^~~~~~~
[ 76%] Linking CXX executable field
[ 76%] Built target field
[ 76%] Linking CXX static library libscream_theta-l_kokkos_4_72_10.a
[ 76%] Built target scream_theta-l_kokkos_4_72_10
[ 76%] Linking CXX executable p3_tests
[ 76%] Built target p3_tests
gmake: *** [Makefile:166: all] Error 2
Error(s) occurred during test phase
OVERALL STATUS: FAIL
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx
weaver failed
######################################################
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins13985744753253938770.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: [email protected]
Finished: FAILURE

Apr 12 '24 00:04 E3SM-Autotester

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5197
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	f5bfe88658d10fd922d247cfa9522f365eb26c0b
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5536
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	f5bfe88658d10fd922d247cfa9522f365eb26c0b
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: mjs271

Apr 12 '24 11:04 E3SM-Autotester

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5197
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	f5bfe88658d10fd922d247cfa9522f365eb26c0b
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5536
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	f5bfe88658d10fd922d247cfa9522f365eb26c0b
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5197 FAILED (click to see last 100 lines of console output)


    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240412_054029_4uc08n
DIFF ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240412_054029_4uc08n
DIFF ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240412_054029_4uc08n
DIFF PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240412_054029_4uc08n
DIFF SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240412_054029_4uc08n
DIFF SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240412_054029_4uc08n
test-scheduler took 795.4733483791351 seconds'
+ errors='Waiting for tests to finish
DIFF ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240412_054029_4uc08n
DIFF ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240412_054029_4uc08n
DIFF ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240412_054029_4uc08n
DIFF ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240412_054029_4uc08n
DIFF ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240412_054029_4uc08n
DIFF PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240412_054029_4uc08n
DIFF SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240412_054029_4uc08n
DIFF SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240412_054029_4uc08n
test-scheduler took 795.4733483791351 seconds'
+ V1_FAILURES_DETAILS+='Waiting for tests to finish
DIFF ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240412_054029_4uc08n
DIFF ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240412_054029_4uc08n
DIFF ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240412_054029_4uc08n
DIFF ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240412_054029_4uc08n
DIFF ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240412_054029_4uc08n
DIFF PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240412_054029_4uc08n
DIFF SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240412_054029_4uc08n
DIFF SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240412_054029_4uc08n
test-scheduler took 795.4733483791351 seconds'
+ set +x
######################################################
FAILS DETECTED:
  SCREAM STANDALONE TESTING FAILED!
Build type full_debug failed at build time. Here's the build log:
Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5197/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m mappy
RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5197/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m mappy
FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5197/scream/components/eamxx
mappy failed
  SCREAM V1 TESTING FAILED!
Waiting for tests to finish
DIFF ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240412_054029_4uc08n
DIFF ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240412_054029_4uc08n
DIFF ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240412_054029_4uc08n
DIFF ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240412_054029_4uc08n
DIFF ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240412_054029_4uc08n
DIFF PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240412_054029_4uc08n
DIFF SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240412_054029_4uc08n
DIFF SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240412_054029_4uc08n
test-scheduler took 795.4733483791351 seconds
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 26381 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins6779381981726991412.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: [email protected]
Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5536 FAILED (click to see last 100 lines of console output)


Warning: Unused dummy argument 'nsplit_iter' at (1) [-Wunused-dummy-argument]
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5536/scream/components/homme/src/theta-l_kokkos/prim_driver_mod.F90:514:8:
514 |     use control_mod, only: test_with_forcing
|        1
Warning: Unused module variable 'test_with_forcing' which has been explicitly imported at (1) [-Wunused-variable]
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5536/scream/components/homme/src/theta-l_kokkos/prim_driver_mod.F90:409:53:
409 |     real(kind=real_kind) :: dt_remap, dt_q, eta_ave_w
|                                                     1
Warning: Unused variable 'eta_ave_w' declared at (1) [-Wunused-variable]
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5536/scream/components/homme/src/theta-l_kokkos/prim_driver_mod.F90:368:74:
368 |   subroutine prim_run_subcycle(elem, hybrid, nets, nete, dt, single_column, tl, hvcoord, nsplit_iteration)
|                                                                          1
Warning: Unused dummy argument 'single_column' at (1) [-Wunused-dummy-argument]
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5536/scream/components/homme/src/theta-l_kokkos/prim_driver_mod.F90:291:40:
291 |   subroutine prim_init_diags_views (elem)
|                                        1
Warning: Unused dummy argument 'elem' at (1) [-Wunused-dummy-argument]
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5536/scream/components/homme/src/theta-l_kokkos/prim_driver_mod.F90:271:45:
271 |   subroutine prim_init_ref_states_views (elem)
|                                             1
Warning: Unused dummy argument 'elem' at (1) [-Wunused-dummy-argument]
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5536/scream/components/homme/src/theta-l_kokkos/prim_driver_mod.F90:240:40:
240 |   subroutine prim_init_state_views (elem)
|                                        1
Warning: Unused dummy argument 'elem' at (1) [-Wunused-dummy-argument]
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5536/scream/components/homme/src/theta-l_kokkos/prim_driver_mod.F90:150:70:
150 |     real (kind=real_kind), target, dimension(np,np)         :: elem_mp, elem_fcor, elem_spheremp
|                                                                      1
Warning: Unused variable 'elem_mp' declared at (1) [-Wunused-variable]
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5536/scream/components/homme/src/theta-l_kokkos/prim_driver_mod.F90:91:17:
91 |     integer :: ie
|                 1
Warning: Unused variable 'ie' declared at (1) [-Wunused-variable]
[ 67%] Linking Fortran static library libtheta-l_kokkos_4_72_41.a
[ 67%] Linking CXX executable p3_tests_fail
[ 67%] Built target theta-l_kokkos_4_72_41
[ 67%] Built target p3_tests_fail
[ 67%] Linking CXX executable io_remap_test
[ 68%] Linking CXX executable io_basic
[ 68%] Linking CXX executable io_monthly
[ 68%] Built target io_remap_test
[ 68%] Linking CXX executable property_checks
[ 68%] Built target io_basic
[ 68%] Built target io_monthly
[ 68%] Linking CXX executable io_packed
[ 68%] Built target property_checks
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5536/scream/externals/ekat/extern/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Team.hpp: In member function 'void Kokkos::Impl::ParallelReduce<FunctorType, Kokkos::TeamPolicy<Properties ...>, ReducerType, Kokkos::Cuda>::execute() [with FunctorType = __nv_hdl_wrapper_t<false, false, __nv_dl_tag<void (*)(), scream::p3::unit_test::UnitWrap::UnitTest<Kokkos::Device<Kokkos::Cuda, Kokkos::CudaSpace> >::TestP3SubgridVarianceScaling::run_property_tests, 1>, void(const Kokkos::Impl::CudaTeamMember&, int&)>; ReducerType = Kokkos::InvalidType; Properties = {Kokkos::Cuda}]':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5536/scream/externals/ekat/extern/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Team.hpp:839:6: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
839 |   inline void execute() {
|      ^~~~~~~
[ 68%] Built target io_packed
[ 69%] Linking CXX executable io_se_grid
[ 69%] Linking CXX executable io_filled
[ 69%] Linking CXX executable io_diags
[ 69%] Built target io_se_grid
[ 69%] Built target io_filled
[ 70%] Linking CXX executable time_interpolation
[ 70%] Built target io_diags
[ 70%] Linking CXX executable atm_proc
[ 70%] Linking CXX executable field_utils
[ 70%] Built target time_interpolation
[ 70%] Built target atm_proc
[ 70%] Built target field_utils
[ 70%] Linking CXX executable output_restart
[ 70%] Built target output_restart
[ 70%] Linking CXX executable field
[ 70%] Built target field
[ 70%] Linking CXX executable p3_tests
[ 70%] Built target p3_tests
gmake: *** [Makefile:166: all] Error 2
Error(s) occurred during test phase
OVERALL STATUS: FAIL
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5536/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5536/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5536/scream/components/eamxx
weaver failed
######################################################
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins3905762627297048643.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: [email protected]
Finished: FAILURE

Apr 12 '24 12:04 E3SM-Autotester

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5200
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	f5bfe88658d10fd922d247cfa9522f365eb26c0b
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5539
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	f5bfe88658d10fd922d247cfa9522f365eb26c0b
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: mjs271

Apr 12 '24 14:04 E3SM-Autotester

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5200
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	f5bfe88658d10fd922d247cfa9522f365eb26c0b
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5539
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	f5bfe88658d10fd922d247cfa9522f365eb26c0b
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5200 FAILED (click to see last 100 lines of console output)


    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240412_083343_0dg4dx
DIFF ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240412_083343_0dg4dx
DIFF ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240412_083343_0dg4dx
DIFF PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240412_083343_0dg4dx
DIFF SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240412_083343_0dg4dx
DIFF SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240412_083343_0dg4dx
test-scheduler took 766.0941200256348 seconds'
+ errors='Waiting for tests to finish
DIFF ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240412_083343_0dg4dx
DIFF ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240412_083343_0dg4dx
DIFF ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240412_083343_0dg4dx
DIFF ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240412_083343_0dg4dx
DIFF ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240412_083343_0dg4dx
DIFF PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240412_083343_0dg4dx
DIFF SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240412_083343_0dg4dx
DIFF SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240412_083343_0dg4dx
test-scheduler took 766.0941200256348 seconds'
+ V1_FAILURES_DETAILS+='Waiting for tests to finish
DIFF ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240412_083343_0dg4dx
DIFF ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240412_083343_0dg4dx
DIFF ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240412_083343_0dg4dx
DIFF ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240412_083343_0dg4dx
DIFF ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240412_083343_0dg4dx
DIFF PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240412_083343_0dg4dx
DIFF SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240412_083343_0dg4dx
DIFF SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240412_083343_0dg4dx
test-scheduler took 766.0941200256348 seconds'
+ set +x
######################################################
FAILS DETECTED:
  SCREAM STANDALONE TESTING FAILED!
Build type full_debug failed at build time. Here's the build log:
Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5200/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m mappy
RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5200/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m mappy
FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5200/scream/components/eamxx
mappy failed
  SCREAM V1 TESTING FAILED!
Waiting for tests to finish
DIFF ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240412_083343_0dg4dx
DIFF ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240412_083343_0dg4dx
DIFF ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240412_083343_0dg4dx
DIFF ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240412_083343_0dg4dx
DIFF ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240412_083343_0dg4dx
DIFF PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240412_083343_0dg4dx
DIFF SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240412_083343_0dg4dx
DIFF SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240412_083343_0dg4dx
test-scheduler took 766.0941200256348 seconds
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 57251 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins9622105594661570145.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: [email protected]
Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5539 FAILED (click to see last 100 lines of console output)


nvcc_internal_extended_lambda_implementation: In static member function 'static void* __nv_hdl_wrapper_t::manager::do_copy(void*) [with Lambda = scream::Field::update_impl<:combinemode::multiply scream::device int>(const scream::Field&, int, int, int)::; Tag = __nv_dl_tag; OpFuncR = void; OpFuncArgs = {int}; F1 = Kokkos::View, Kokkos::MemoryTraits >; F2 = Kokkos::View, Kokkos::MemoryTraits >; F3 = const int; F4 = const int; F5 = const int]':
nvcc_internal_extended_lambda_implementation:463:19: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
[ 64%] Built target coarsening_remapper
[ 64%] Linking CXX executable physics_test_data
[ 64%] Built target grid
[ 64%] Linking CXX executable physics_saturation_run_and_cmp
[ 64%] Building Fortran object src/physics/tms/tests/CMakeFiles/tms_tests.dir/__/tms_iso_c.f90.o
[ 64%] Linking CXX executable common_physics
[ 64%] Built target physics_test_data
[ 64%] Built target physics_saturation_run_and_cmp
[ 64%] Linking CXX executable io_monthly
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5539/scream/externals/ekat/extern/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Team.hpp: In constructor 'Kokkos::Impl::ParallelFor, Kokkos::Cuda>::ParallelFor(const FunctorType&, const Policy&) [with FunctorType = __nv_hdl_wrapper_t&), scream::run<:device kokkos::cudaspace> >, 1>, void(const Kokkos::Impl::CudaTeamMember&), const int, const Kokkos::View<:pack>**, Kokkos::LayoutRight, Kokkos::Device<:cuda kokkos::cudaspace>, Kokkos::MemoryTraits >, const Kokkos::View<:pack>**, Kokkos::LayoutRight, Kokkos::Device<:cuda kokkos::cudaspace>, Kokkos::MemoryTraits > >; Properties = {Kokkos::Cuda}]':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5539/scream/externals/ekat/extern/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Team.hpp:559:1: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  559 |   ParallelFor(const FunctorType& arg_functor, const Policy& arg_policy)
      | ^ ~~~~~~~~~
[ 64%] Built target common_physics
[ 64%] Linking CXX executable io_remap_test
[ 65%] Linking CXX executable io_basic
[ 65%] Linking CXX executable io_packed
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5539/scream/externals/ekat/extern/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Team.hpp: In constructor 'Kokkos::Impl::ParallelFor, Kokkos::Cuda>::ParallelFor(const FunctorType&, const Policy&) [with FunctorType = __nv_hdl_wrapper_t&), scream::run<:device kokkos::cudaspace> >, 1>, void(const Kokkos::Impl::CudaTeamMember&), const int, const Kokkos::View<:pack>**, Kokkos::LayoutRight, Kokkos::Device<:cuda kokkos::cudaspace>, Kokkos::MemoryTraits >, const Kokkos::View<:pack>**, Kokkos::LayoutRight, Kokkos::Device<:cuda kokkos::cudaspace>, Kokkos::MemoryTraits >, const Kokkos::View<:pack>**, Kokkos::LayoutRight, Kokkos::Device<:cuda kokkos::cudaspace>, Kokkos::MemoryTraits > >; Properties = {Kokkos::Cuda}]':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5539/scream/externals/ekat/extern/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Team.hpp:559:1: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  559 |   ParallelFor(const FunctorType& arg_functor, const Policy& arg_policy)
      | ^ ~~~~~~~~~
[ 65%] Linking CXX executable tms_tests
[ 65%] Built target io_monthly
[ 65%] Linking CXX static library libp3.a
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5539/scream/externals/ekat/extern/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Team.hpp: In constructor 'Kokkos::Impl::ParallelFor, Kokkos::Cuda>::ParallelFor(const FunctorType&, const Policy&) [with FunctorType = __nv_hdl_wrapper_t&), scream::run<:device kokkos::cudaspace> >, 1>, void(const Kokkos::Impl::CudaTeamMember&), const int, const Kokkos::View<:pack>**, Kokkos::LayoutRight, Kokkos::Device<:cuda kokkos::cudaspace>, Kokkos::MemoryTraits >, const Kokkos::View<:pack>**, Kokkos::LayoutRight, Kokkos::Device<:cuda kokkos::cudaspace>, Kokkos::MemoryTraits >, const Kokkos::View<:pack>**, Kokkos::LayoutRight, Kokkos::Device<:cuda kokkos::cudaspace>, Kokkos::MemoryTraits > >; Properties = {Kokkos::Cuda}]':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5539/scream/externals/ekat/extern/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Team.hpp:559:1: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  559 |   ParallelFor(const FunctorType& arg_functor, const Policy& arg_policy)
      | ^ ~~~~~~~~~
[ 65%] Built target io_basic
[ 65%] Built target io_remap_test
[ 65%] Built target p3
[ 65%] Built target io_packed
[ 65%] Built target tms_tests
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5539/scream/externals/ekat/extern/kokkos/core/src/KokkosExp_MDRangePolicy.hpp: In copy constructor 'Kokkos::MDRangePolicy<:cuda kokkos::rank kokkos::iterate::right>, Kokkos::IndexType >::MDRangePolicy(const Kokkos::MDRangePolicy<:cuda kokkos::rank kokkos::iterate::right>, Kokkos::IndexType >&)':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5539/scream/externals/ekat/extern/kokkos/core/src/KokkosExp_MDRangePolicy.hpp:185:8: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  185 | struct MDRangePolicy : public Kokkos::Impl::PolicyTraits {
      |        ^~~~~~~~~~~~~
[ 65%] Linking CXX executable property_checks
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5539/scream/externals/ekat/extern/kokkos/core/src/KokkosExp_MDRangePolicy.hpp: In copy constructor 'Kokkos::MDRangePolicy<:cuda kokkos::rank kokkos::iterate::right>, Kokkos::IndexType >::MDRangePolicy(const Kokkos::MDRangePolicy<:cuda kokkos::rank kokkos::iterate::right>, Kokkos::IndexType >&)':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5539/scream/externals/ekat/extern/kokkos/core/src/KokkosExp_MDRangePolicy.hpp:185:8: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  185 | struct MDRangePolicy : public Kokkos::Impl::PolicyTraits {
      |        ^~~~~~~~~~~~~
[ 65%] Built target property_checks
[ 65%] Linking CXX executable nudging_data
[ 65%] Linking CXX executable field_at_pressure_level
[ 65%] Linking CXX executable field_at_level
[ 65%] Linking CXX executable spa_main_test
[ 66%] Linking CXX executable io_se_grid
[ 66%] Linking CXX executable io_filled
[ 66%] Built target nudging_data
[ 66%] Built target field_at_pressure_level
[ 66%] Built target field_at_level
[ 66%] Built target spa_main_test
[ 66%] Built target io_filled
[ 66%] Built target io_se_grid
[ 66%] Linking CXX executable io_diags
[ 66%] Linking CXX executable field_utils
[ 66%] Built target io_diags
[ 66%] Built target field_utils
[ 67%] Linking CXX executable field_at_height
[ 67%] Linking CXX executable output_restart
[ 67%] Linking CXX executable exner_function
[ 67%] Built target field_at_height
[ 67%] Linking CXX executable virtual_temperature
[ 67%] Built target output_restart
[ 67%] Built target exner_function
[ 67%] Built target virtual_temperature
[ 68%] Linking CXX executable time_interpolation
[ 68%] Linking CXX executable potential_temperature
[ 68%] Built target time_interpolation
[ 68%] Linking CXX executable atm_proc
[ 68%] Built target potential_temperature
[ 68%] Built target atm_proc
[ 68%] Linking CXX executable field
[ 68%] Built target field
gmake: *** [Makefile:166: all] Error 2
Error(s) occurred during test phase
OVERALL STATUS: FAIL
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5539/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5539/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5539/scream/components/eamxx
weaver failed
######################################################
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins9407770862716668100.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: [email protected]
Finished: FAILURE

Apr 12 '24 14:04 E3SM-Autotester

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5203
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	f5bfe88658d10fd922d247cfa9522f365eb26c0b
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5542
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	f5bfe88658d10fd922d247cfa9522f365eb26c0b
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: mjs271

Apr 12 '24 17:04 E3SM-Autotester

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5203
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	f5bfe88658d10fd922d247cfa9522f365eb26c0b
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5542
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	f5bfe88658d10fd922d247cfa9522f365eb26c0b
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5203 FAILED (click to see last 100 lines of console output)


[ 78%] Linking CXX executable p3_tests_fail
[ 78%] Built target vertical_remapper
[ 78%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_ice_sed_unit_tests.cpp.o
[ 78%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_ice_collection_unit_tests.cpp.o
[ 78%] Built target scream_theta-l_kokkos_4_128_10
[ 78%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_rain_sed_unit_tests.cpp.o
[ 78%] Built target refining_remapper_p2p
[ 78%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_dsd2_unit_tests.cpp.o
[ 78%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_rain_self_collection_tests.cpp.o
[ 78%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_autoconversion_unit_tests.cpp.o
[ 78%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_ice_relaxation_timescale_unit_tests.cpp.o
[ 78%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_calc_liq_relaxation_timescale_unit_tests.cpp.o
[ 78%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_ice_nucleation_unit_tests.cpp.o
[ 78%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_ice_melting_unit_tests.cpp.o
[ 78%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_evaporate_rain_unit_tests.cpp.o
[ 78%] Built target coarsening_remapper
[ 78%] Built target p3_tests_fail
[ 79%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_ice_cldliq_wet_growth_unit_tests.cpp.o
[ 79%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_get_latent_heat_unit_tests.cpp.o
[ 79%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_subgrid_variance_scaling_unit_tests.cpp.o
[ 79%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_check_values_unit_tests.cpp.o
[ 79%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_incloud_mixingratios_unit_tests.cpp.o
[ 79%] Built target grid
[ 79%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_main_unit_tests.cpp.o
[ 79%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_ice_supersat_conservation_tests.cpp.o
[ 79%] Built target io_basic
[ 79%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_nc_conservation_tests.cpp.o
[ 79%] Built target io_monthly
[ 79%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_nr_conservation_tests.cpp.o
[ 79%] Linking CXX executable io_filled
[ 79%] Linking CXX executable io_se_grid
[ 79%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_ni_conservation_tests.cpp.o
[ 79%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_ice_deposition_sublimation_tests.cpp.o
[ 79%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_prevent_liq_supersaturation_tests.cpp.o
[ 79%] Linking CXX executable property_checks
[ 79%] Linking CXX executable io_diags
[ 79%] Built target io_packed
[ 80%] Linking CXX static library libscream_control.a
[ 80%] Built target io_remap_test
[ 80%] Built target property_checks
[ 80%] Built target scream_control
[ 80%] Linking CXX executable p3_tests
[ 80%] Linking CXX executable field_utils
[ 80%] Linking CXX executable atm_proc
[ 80%] Linking CXX executable output_restart
[ 80%] Built target io_filled
[ 80%] Built target io_se_grid
[ 80%] Built target io_diags
[ 80%] Built target p3_tests
[ 80%] Built target field_utils
[ 80%] Linking CXX executable time_interpolation
[ 80%] Built target atm_proc
[ 80%] Built target output_restart
[ 80%] Built target time_interpolation
[ 80%] Linking CXX executable field
[ 80%] Built target field
gmake: *** [all] Error 2
  SCREAM V1 TESTING FAILED!
Waiting for tests to finish
DIFF ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240412_113233_z57cqt
DIFF ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240412_113233_z57cqt
DIFF ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240412_113233_z57cqt
DIFF ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240412_113233_z57cqt
DIFF ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240412_113233_z57cqt
DIFF PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240412_113233_z57cqt
DIFF SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240412_113233_z57cqt
DIFF SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240412_113233_z57cqt
test-scheduler took 787.4738502502441 seconds
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 9627 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins9509961460591995187.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: [email protected]
Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5542 FAILED (click to see last 100 lines of console output)


  990 |   Kokkos::parallel_for(Kokkos::RangePolicy(0,nelem*qsize*NGP*NGP*npacks),
      |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ^
[ 70%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_ice_supersat_conservation_tests.cpp.o
[ 70%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_nc_conservation_tests.cpp.o
nvcc_internal_extended_lambda_implementation: In static member function 'static void* __nv_hdl_wrapper_t::manager::do_copy(void*) [with Lambda = scream::Field::update_impl<:combinemode::multiply scream::device int>(const scream::Field&, int, int, int)::; Tag = __nv_dl_tag; OpFuncR = void; OpFuncArgs = {int}; F1 = Kokkos::View, Kokkos::MemoryTraits >; F2 = Kokkos::View, Kokkos::MemoryTraits >; F3 = const int; F4 = const int; F5 = const int]':
nvcc_internal_extended_lambda_implementation:463:19: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5542/scream/externals/ekat/extern/kokkos/core/src/KokkosExp_MDRangePolicy.hpp: In copy constructor 'Kokkos::MDRangePolicy<:cuda kokkos::rank kokkos::iterate::right>, Kokkos::IndexType >::MDRangePolicy(const Kokkos::MDRangePolicy<:cuda kokkos::rank kokkos::iterate::right>, Kokkos::IndexType >&)':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5542/scream/externals/ekat/extern/kokkos/core/src/KokkosExp_MDRangePolicy.hpp:185:8: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  185 | struct MDRangePolicy : public Kokkos::Impl::PolicyTraits {
      |        ^~~~~~~~~~~~~
[ 70%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_nr_conservation_tests.cpp.o
[ 70%] Built target grid
[ 70%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_ni_conservation_tests.cpp.o
[ 70%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_ice_deposition_sublimation_tests.cpp.o
[ 70%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_prevent_liq_supersaturation_tests.cpp.o
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5542/scream/externals/ekat/extern/kokkos/core/src/KokkosExp_MDRangePolicy.hpp: In copy constructor 'Kokkos::MDRangePolicy<:cuda kokkos::rank kokkos::iterate::right>, Kokkos::IndexType >::MDRangePolicy(const Kokkos::MDRangePolicy<:cuda kokkos::rank kokkos::iterate::right>, Kokkos::IndexType >&)':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5542/scream/externals/ekat/extern/kokkos/core/src/KokkosExp_MDRangePolicy.hpp:185:8: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  185 | struct MDRangePolicy : public Kokkos::Impl::PolicyTraits {
      |        ^~~~~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5542/scream/externals/ekat/extern/kokkos/core/src/KokkosExp_MDRangePolicy.hpp: In copy constructor 'Kokkos::MDRangePolicy<:cuda kokkos::rank kokkos::iterate::right>, Kokkos::IndexType >::MDRangePolicy(const Kokkos::MDRangePolicy<:cuda kokkos::rank kokkos::iterate::right>, Kokkos::IndexType >&)':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5542/scream/externals/ekat/extern/kokkos/core/src/KokkosExp_MDRangePolicy.hpp:185:8: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  185 | struct MDRangePolicy : public Kokkos::Impl::PolicyTraits {
      |        ^~~~~~~~~~~~~
[ 71%] Linking CXX executable io_basic
[ 71%] Built target io_basic
[ 71%] Linking CXX executable io_monthly
[ 71%] Linking CXX executable io_packed
[ 71%] Built target io_monthly
[ 71%] Linking CXX executable io_remap_test
[ 71%] Linking CXX executable p3_tests_fail
[ 71%] Built target io_packed
[ 71%] Built target p3_tests_fail
[ 71%] Linking CXX executable property_checks
[ 71%] Built target io_remap_test
/projects/ppc64le-pwr9-rhel8/compilers/gcc/11.3.0/gcc/8.3.1/base/tchbki3/lib/gcc/powerpc64le-unknown-linux-gnu/11.3.0/../../../../include/c++/11.3.0/bits/stl_pair.h: In instantiation of 'constexpr std::pair::type>::__type, typename std::__strip_reference_wrapper::type>::__type> std::make_pair(_T1&&, _T2&&) [with _T1 = double; _T2 = double; typename std::__strip_reference_wrapper::type>::__type = double; typename std::decay<_tp2>::type = double; typename std::__strip_reference_wrapper::type>::__type = double; typename std::decay<_tp>::type = double]':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5542/scream/components/eamxx/src/physics/p3/tests/p3_ice_sed_unit_tests.cpp:204:37:   required from 'static void scream::p3::unit_test::UnitWrap::UnitTest::TestIceSed::run_bfb_homogeneous_freezing() [with D = Kokkos::Device<:cuda kokkos::cudaspace>]'
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5542/scream/components/eamxx/src/physics/p3/tests/p3_ice_sed_unit_tests.cpp:253:31:   required from 'static void scream::p3::unit_test::UnitWrap::UnitTest::TestIceSed::run_bfb() [with D = Kokkos::Device<:cuda kokkos::cudaspace>]'
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5542/scream/components/eamxx/src/physics/p3/tests/p3_ice_sed_unit_tests.cpp:269:6:   required from here
/projects/ppc64le-pwr9-rhel8/compilers/gcc/11.3.0/gcc/8.3.1/base/tchbki3/lib/gcc/powerpc64le-unknown-linux-gnu/11.3.0/../../../../include/c++/11.3.0/bits/stl_pair.h:567:1: note: parameter passing for argument of type 'std::pair' when C++17 is enabled changed to match C++14 in GCC 10.1
  567 |     make_pair(_T1&& __x, _T2&& __y)
      | ^   ~~~~~
[ 71%] Built target property_checks
[ 71%] Linking CXX executable field_utils
[ 71%] Linking CXX executable io_filled
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5542/scream/externals/ekat/extern/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Range.hpp: In member function 'void Kokkos::Impl::ParallelReduce, ReducerType, Kokkos::Cuda>::execute() [with FunctorType = __nv_hdl_wrapper_t >::TestP3CloudWaterAutoconversion::run_physics, 1>, void(const int&, int&)>; ReducerType = Kokkos::InvalidType; Traits = {Kokkos::Cuda}]':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5542/scream/externals/ekat/extern/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Range.hpp:321:6: note: the layout of aggregates containing vectors with 2-byte alignment has changed in GCC 5
  321 |   inline void execute() {
      |      ^~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5542/scream/components/eamxx/src/physics/p3/tests/p3_ice_nucleation_unit_tests.cpp: In static member function 'static void* __nv_hdl_wrapper_t::manager::do_copy(void*) [with Lambda = scream::p3::unit_test::UnitWrap::UnitTest<:device kokkos::cudaspace> >::TestIceNucleation::run_ice_nucleation_bfb()::; Tag = __nv_dl_tag >::TestIceNucleation::run_ice_nucleation_bfb, 1>; OpFuncR = void; OpFuncArgs = {const int&}; F1 = Kokkos::View<:p3::icenucleationdata kokkos::layoutright kokkos::device kokkos::cudaspace>, Kokkos::MemoryTraits >; F2 = bool; F3 = bool]':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5542/scream/components/eamxx/src/physics/p3/tests/p3_ice_nucleation_unit_tests.cpp:69:348: note: the layout of aggregates containing vectors with 2-byte alignment has changed in GCC 5
   69 |         Kokkos::parallel_for(num_test_itrs, KOKKOS_LAMBDA(const Int& i) {
[ 72%] Linking CXX executable io_se_grid
[ 72%] Built target field_utils
[ 72%] Built target io_filled
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5542/scream/externals/ekat/extern/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Team.hpp: In member function 'void Kokkos::Impl::ParallelReduce, ReducerType, Kokkos::Cuda>::execute() [with FunctorType = __nv_hdl_wrapper_t >::TestP3SubgridVarianceScaling::run_property_tests, 1>, void(const Kokkos::Impl::CudaTeamMember&, int&)>; ReducerType = Kokkos::InvalidType; Properties = {Kokkos::Cuda}]':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5542/scream/externals/ekat/extern/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Team.hpp:839:6: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  839 |   inline void execute() {
      |      ^~~~~~~
[ 72%] Linking CXX executable io_diags
[ 72%] Built target io_se_grid
[ 73%] Linking CXX executable time_interpolation
[ 74%] Linking CXX static library libscream_theta-l_kokkos_4_72_41.a
[ 74%] Built target io_diags
[ 74%] Linking CXX executable atm_proc
[ 74%] Built target scream_theta-l_kokkos_4_72_41
[ 74%] Linking CXX executable output_restart
[ 74%] Built target time_interpolation
[ 74%] Built target atm_proc
[ 74%] Built target output_restart
[ 74%] Linking CXX static library libscream_theta-l_kokkos_4_128_10.a
[ 74%] Built target scream_theta-l_kokkos_4_128_10
[ 74%] Linking CXX static library libscream_theta-l_kokkos_4_72_10.a
[ 74%] Built target scream_theta-l_kokkos_4_72_10
[ 74%] Linking CXX executable field
[ 74%] Built target field
[ 74%] Linking CXX executable p3_tests
[ 74%] Built target p3_tests
gmake: *** [Makefile:166: all] Error 2
Error(s) occurred during test phase
OVERALL STATUS: FAIL
Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5542/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5542/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver
FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5542/scream/components/eamxx
weaver failed
######################################################
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins13487919682087876277.sh
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Sending e-mails to: [email protected]
Finished: FAILURE

Apr 12 '24 17:04 E3SM-Autotester

@mjs271 , it looks like this is failing the weaver test suite for the AT. Can you take a look?

Apr 16 '24 16:04 AaronDonahue

@AaronDonahue I'll give it a look!

Apr 16 '24 16:04 mjs271

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5218
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	7929a16dabc93457d73621e0b8fa72e2d33418ab
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5557
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	7929a16dabc93457d73621e0b8fa72e2d33418ab
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: mjs271

Apr 16 '24 22:04 E3SM-Autotester

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5218
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	7929a16dabc93457d73621e0b8fa72e2d33418ab
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5557
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	7929a16dabc93457d73621e0b8fa72e2d33418ab
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Apr 16 '24 22:04 E3SM-Autotester

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging THE LAST COMMIT TO THIS PULL REQUEST HAS BEEN REVIEWED, BUT NOT ACCEPTED OR REQUIRES CHANGES!

Apr 16 '24 22:04 E3SM-Autotester

All Jobs Finished; status = PASSED, target_sha=17db7f0c567358af89ddfdc2a75e060000f80767, However Inspection must be performed before merge can occur...

Apr 16 '24 22:04 E3SM-Autotester

The base branch has been updated since the last successful testing.

last PASS base branch sha: 17db7f0c567358af89ddfdc2a75e060000f80767
current base branch sha : 6137460df75df123dbffe471c39a019521d0d4b2 The AutoTester will discard the last PASS, and re-test the PR from scratch

Apr 17 '24 03:04 E3SM-Autotester

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5227
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	7929a16dabc93457d73621e0b8fa72e2d33418ab
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5562
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	7929a16dabc93457d73621e0b8fa72e2d33418ab
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	a6c73f47ede28cf44f6610a7146c5ecb96f7202f
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: mjs271

Apr 17 '24 14:04 E3SM-Autotester

scream scream copied to clipboard

Enable Subfield Slices Spanning a Range of indices (Issue #1312)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Using Repos:

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

Total Test time (real) = 779.67 sec

Testing '''16d63ec1f1e2a1d356d2faf4d8bdd8cf3fbbe560''' for test '''release'''

Testing '''16d63ec1f1e2a1d356d2faf4d8bdd8cf3fbbe560''' for test '''full_sp_debug'''

Testing '''16d63ec1f1e2a1d356d2faf4d8bdd8cf3fbbe560''' for test '''full_debug'''

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Using Repos:

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

Total Test time (real) = 582.95 sec

Testing '''d9c0bc846ffc60d14b68586182183bfa9a3c359d''' for test '''full_sp_debug'''

Testing '''d9c0bc846ffc60d14b68586182183bfa9a3c359d''' for test '''release'''

Testing '''d9c0bc846ffc60d14b68586182183bfa9a3c359d''' for test '''full_debug'''

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Using Repos:

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

Total Test time (real) = 647.87 sec

Testing '''50f995e8afe99f8bbef9242a1f3806095ed19f17''' for test '''release'''

Testing '''50f995e8afe99f8bbef9242a1f3806095ed19f17''' for test '''full_sp_debug'''

Testing '''50f995e8afe99f8bbef9242a1f3806095ed19f17''' for test '''full_debug'''

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Using Repos:

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

Total Test time (real) = 711.15 sec

Testing '''a32dba4fdd314a7622254a344dcb94e1cb631ca7''' for test '''full_sp_debug'''

Testing '''a32dba4fdd314a7622254a344dcb94e1cb631ca7''' for test '''release'''

Testing '''a32dba4fdd314a7622254a344dcb94e1cb631ca7''' for test '''full_debug'''

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

scream
scream copied to clipboard