ROL tests failing in ATDM Trilinos builds starting 07-21-2020

[e10harvey]

CC: @trilinos/rol, @rppawlo, @jwillenbring, @bartlettroscoe

Next Action Status

Description

As shown in this query (click "Shown Matching Output" in upper right) the tests:

• ROL_example_PDE-OPT_stefan-boltzmann_example_01_MPI_4
• ROL_example_tempus_example_02_MPI_1
• ROL_test_sol_checkSuperQuantileQuadrangle_MPI_1
• ROL_test_vector_StdArrayInterface_MPI_1

in the builds:

• Trilinos-atdm-ats1-knl_intel-19.0.4_mpich-7.7.15_openmp_static_opt
• Trilinos-atdm-ats1-knl_intel-19.0.4_mpich-7.7.6_openmp_static_opt
• Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_serial_static_dbg
• Trilinos-atdm-cts1-intel-19.0.4_openmpi-4.0.3_openmp_static_dbg
• Trilinos-atdm-cts1-intel-19.0.4_openmpi-4.0.3_openmp_static_opt
• Trilinos-atdm-van1-tx2_arm-20.0_openmpi-4.0.2_openmp_static_dbg
• Trilinos-atdm-van1-tx2_arm-20.1_openmpi-4.0.3_openmp_static_dbg

started failing on testing day 2020-07-22.

Related to https://github.com/trilinos/Trilinos/issues/8486 and https://github.com/trilinos/Trilinos/issues/6553.

As shown in https://testing-dev.sandia.gov/cdash/test/41294765, https://testing-dev.sandia.gov/cdash/test/41266187, https://testing-dev.sandia.gov/cdash/test/25915010, and https://testing-dev.sandia.gov/cdash/test/28236759 we are seeing errors like:

********** Begin verification of linear algebra. *********************************************

Commutativity of addition. Consistency error: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0.000000000000e+00
Associativity of addition. Consistency error: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2.719479911021e-16
Identity element of addition. Consistency error: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0.000000000000e+00
Inverse elements of addition. Consistency error: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0.000000000000e+00
Identity element of scalar multiplication. Consistency error: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0.000000000000e+00
Consistency of scalar multiplication with field multiplication. Consistency error: >>>>>>>>>>> 2.623306862686e-16
Distributivity of scalar multiplication with respect to field addition. Consistency error: >>> 5.644999164097e-16
Distributivity of scalar multiplication with respect to vector addition. Consistency error: >> 7.485252198913e-16
Commutativity of dot (inner) product over the field of reals. Consistency error: >>>>>>>>>>>>> 0.000000000000e+00
Additivity of dot (inner) product. Consistency error: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0.000000000000e+00
Consistency of scalar multiplication and norm. Consistency error: >>>>>>>>>>>>>>>>>>>>>>>>>>>> 0.000000000000e+00
Reflexivity. Consistency error: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 0.000000000000e+00

Current Status on CDash

Run the above query adjusting the "Begin" and "End" dates to match today any other date range or just click "CURRENT" in the top bar to see results for the current testing day.

Steps to Reproduce

One should be able to reproduce this failure as described in:

• https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md

and the system-specific instructions at:

• https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#specific-instructions-for-each-system

Just log into any of the associated machines and copy and paste the full CDash build name <build-name> listed above and run commands like:

$ cd <some_build_dir>/

$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh <build-name>

$ cmake \
 -GNinja \
 -DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
 -DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_<package-name>=ON \
 $TRILINOS_DIR

$ make NP=16

$ <command-to-run-on-compute-node> ctest -j4

where <package-name> is any package that you want to enable to reproduce build and/or test results.

Again, for exact system-specific details on what commands to run to build and run tests, see:

• https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#specific-instructions-for-each-system

If you can't figure out what commands to run to reproduce the problem given this documentation, then please post a comment here and we will give you the exact minimal commands.

[dridzal]

There is a major ROL update (ROL-v2.0) coming in the next two-three weeks. This will likely change the behavior of most tests and examples. I suggest we wait until that is completed.

[grover-trilinos]

Test results for issue #8613 as of 2021-01-24

Tests with issue trackers Failed: twif=3

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=3

Site	Build Name	Test Name	Status	Details	Consec­utive Non-pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
mutrino	Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	14	14	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	26	26	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	30	30	0	#8613

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #8613 as of 2021-01-31

Tests with issue trackers Passed: twip=2
Tests with issue trackers Failed: twif=5

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=2

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­example_­PDE-OPT_­stefan-boltzmann_­example_­01_­MPI_­4	Passed	Completed	24	0	24	#8613
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­test_­sol_­checkSuperQuantileQuadrangle_­MPI_­1	Passed	Completed	24	0	24	#8613

Tests with issue trackers Failed: twif=5

Site	Build Name	Test Name	Status	Details	Consec­utive Non-pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
mutrino	Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	20	20	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	26	26	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	28	28	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	27	27	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	27	27	0	#8613

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[e10harvey]

@rppawlo: Who is the assignee for this issue?

[rppawlo]

@e10harvey - assigning to @dridzal

[grover-trilinos]

Test results for issue #8613 as of 2021-02-07

Tests with issue trackers Failed: twif=5

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=5

Site	Build Name	Test Name	Status	Details	Consec­utive Non-pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
mutrino	Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	27	27	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	27	27	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	28	28	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	26	26	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	26	26	0	#8613

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #8613 as of 2021-02-14

Tests with issue trackers Passed: twip=2
Tests with issue trackers Failed: twif=4

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=2

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­example_­PDE-OPT_­stefan-boltzmann_­example_­01_­MPI_­4	Passed	Completed	25	0	25	#8613
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­test_­sol_­checkSuperQuantileQuadrangle_­MPI_­1	Passed	Completed	25	0	25	#8613

Tests with issue trackers Failed: twif=4

Site	Build Name	Test Name	Status	Details	Consec­utive Non-pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	27	27	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	28	28	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	26	26	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	26	26	0	#8613

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #8613 as of 2021-02-21

Tests with issue trackers Passed: twip=2
Tests with issue trackers Failed: twif=5

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=2

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­example_­PDE-OPT_­stefan-boltzmann_­example_­01_­MPI_­4	Passed	Completed	26	0	26	#8613
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­test_­sol_­checkSuperQuantileQuadrangle_­MPI_­1	Passed	Completed	26	0	26	#8613

Tests with issue trackers Failed: twif=5

Site	Build Name	Test Name	Status	Details	Consec­utive Non-pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
mutrino	Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	24	24	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	28	28	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	28	28	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	25	25	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	24	24	0	#8613

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[e10harvey]

Thanks, @dridzal. Any updates on this?

[grover-trilinos]

Test results for issue #8613 as of 2021-02-28

Tests with issue trackers Passed: twip=2
Tests with issue trackers Failed: twif=3

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=2

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­example_­PDE-OPT_­stefan-boltzmann_­example_­01_­MPI_­4	Passed	Completed	28	0	28	#8613
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­test_­sol_­checkSuperQuantileQuadrangle_­MPI_­1	Passed	Completed	28	0	28	#8613

Tests with issue trackers Failed: twif=3

Site	Build Name	Test Name	Status	Details	Consec­utive Non-pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
mutrino	Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	25	25	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	30	30	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	30	30	0	#8613

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #8613 as of 2021-03-07

Tests with issue trackers Passed: twip=2
Tests with issue trackers Failed: twif=3

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=2

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­example_­PDE-OPT_­stefan-boltzmann_­example_­01_­MPI_­4	Passed	Completed	27	0	27	#8613
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­test_­sol_­checkSuperQuantileQuadrangle_­MPI_­1	Passed	Completed	27	0	27	#8613

Tests with issue trackers Failed: twif=3

Site	Build Name	Test Name	Status	Details	Consec­utive Non-pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
mutrino	Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	25	25	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	30	30	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	30	30	0	#8613

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #8613 as of 2021-03-14

Tests with issue trackers Passed: twip=2
Tests with issue trackers Failed: twif=5

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=2

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­example_­PDE-OPT_­stefan-boltzmann_­example_­01_­MPI_­4	Passed	Completed	29	0	29	#8613
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­test_­sol_­checkSuperQuantileQuadrangle_­MPI_­1	Passed	Completed	29	0	29	#8613

Tests with issue trackers Failed: twif=5

Site	Build Name	Test Name	Status	Details	Consec­utive Non-pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
mutrino	Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	26	26	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	30	30	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	30	30	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	16	16	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	14	14	0	#8613

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #8613 as of 2021-03-21

Tests with issue trackers Passed: twip=2
Tests with issue trackers Failed: twif=3

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=2

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­example_­PDE-OPT_­stefan-boltzmann_­example_­01_­MPI_­4	Passed	Completed	29	0	29	#8613
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­test_­sol_­checkSuperQuantileQuadrangle_­MPI_­1	Passed	Completed	29	0	29	#8613

Tests with issue trackers Failed: twif=3

Site	Build Name	Test Name	Status	Details	Consec­utive Non-pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
mutrino	Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	30	30	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	18	18	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	17	17	0	#8613

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #8613 as of 2021-03-28

Tests with issue trackers Passed: twip=2
Tests with issue trackers Failed: twif=5

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=2

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­example_­PDE-OPT_­stefan-boltzmann_­example_­01_­MPI_­4	Passed	Completed	29	0	29	#8613
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­test_­sol_­checkSuperQuantileQuadrangle_­MPI_­1	Passed	Completed	29	0	29	#8613

Tests with issue trackers Failed: twif=5

Site	Build Name	Test Name	Status	Details	Consec­utive Non-pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
mutrino	Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	28	28	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	27	27	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	27	27	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	20	20	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	20	20	0	#8613

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #8613 as of 2021-04-04

Tests with issue trackers Passed: twip=2
Tests with issue trackers Failed: twif=5

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=2

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­example_­PDE-OPT_­stefan-boltzmann_­example_­01_­MPI_­4	Passed	Completed	30	0	30	#8613
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­test_­sol_­checkSuperQuantileQuadrangle_­MPI_­1	Passed	Completed	30	0	30	#8613

Tests with issue trackers Failed: twif=5

Site	Build Name	Test Name	Status	Details	Consec­utive Non-pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
mutrino	Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	28	28	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	27	27	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	27	27	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	27	27	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	27	27	0	#8613

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #8613 as of 2021-04-11

Tests with issue trackers Passed: twip=2
Tests with issue trackers Failed: twif=5

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=2

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­example_­PDE-OPT_­stefan-boltzmann_­example_­01_­MPI_­4	Passed	Completed	30	0	30	#8613
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­test_­sol_­checkSuperQuantileQuadrangle_­MPI_­1	Passed	Completed	30	0	30	#8613

Tests with issue trackers Failed: twif=5

Site	Build Name	Test Name	Status	Details	Consec­utive Non-pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
mutrino	Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	28	28	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	27	27	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	27	27	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	30	30	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	30	30	0	#8613

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #8613 as of 2021-04-18

Tests with issue trackers Passed: twip=2
Tests with issue trackers Failed: twif=5

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=2

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­example_­PDE-OPT_­stefan-boltzmann_­example_­01_­MPI_­4	Passed	Completed	30	0	30	#8613
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­test_­sol_­checkSuperQuantileQuadrangle_­MPI_­1	Passed	Completed	30	0	30	#8613

Tests with issue trackers Failed: twif=5

Site	Build Name	Test Name	Status	Details	Consec­utive Non-pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
mutrino	Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	28	28	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	28	28	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	28	28	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	30	30	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	30	30	0	#8613

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #8613 as of 2021-04-25

Tests with issue trackers Passed: twip=2
Tests with issue trackers Failed: twif=4

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=2

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­example_­PDE-OPT_­stefan-boltzmann_­example_­01_­MPI_­4	Passed	Completed	30	0	30	#8613
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­test_­sol_­checkSuperQuantileQuadrangle_­MPI_­1	Passed	Completed	30	0	30	#8613

Tests with issue trackers Failed: twif=4

Site	Build Name	Test Name	Status	Details	Consec­utive Non-pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	30	30	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	30	30	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	30	30	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	30	30	0	#8613

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #8613 as of 2021-05-02

Tests with issue trackers Passed: twip=2
Tests with issue trackers Failed: twif=5

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=2

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­example_­PDE-OPT_­stefan-boltzmann_­example_­01_­MPI_­4	Passed	Completed	30	0	30	#8613
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­test_­sol_­checkSuperQuantileQuadrangle_­MPI_­1	Passed	Completed	30	0	30	#8613

Tests with issue trackers Failed: twif=5

Site	Build Name	Test Name	Status	Details	Consec­utive Non-pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
mutrino	Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	27	27	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	30	30	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	30	30	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	30	30	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	30	30	0	#8613

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #8613 as of 2021-05-09

Tests with issue trackers Passed: twip=2
Tests with issue trackers Failed: twif=4

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=2

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­example_­PDE-OPT_­stefan-boltzmann_­example_­01_­MPI_­4	Passed	Completed	30	0	30	#8613
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­test_­sol_­checkSuperQuantileQuadrangle_­MPI_­1	Passed	Completed	30	0	30	#8613

Tests with issue trackers Failed: twif=4

Site	Build Name	Test Name	Status	Details	Consec­utive Non-pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
mutrino	Trilinos-atdm-ats1-knl_­intel-19.0.4_­mpich-7.7.15_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	27	27	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	30	30	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	30	30	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	30	30	0	#8613

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[grover-trilinos]

Test results for issue #8613 as of 2021-05-23

Tests with issue trackers Failed: twif=2

Detailed test results: (click to expand)

Tests with issue trackers Failed: twif=2

Site	Build Name	Test Name	Status	Details	Consec­utive Non-pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	19	19	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	29	29	0	#8613

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[ccober6]

Hi @dridzal, can look at these also? Thanks!

[grover-trilinos]

Test results for issue #8613 as of 2021-05-30

Tests with issue trackers Passed: twip=2
Tests with issue trackers Failed: twif=4

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=2

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­example_­PDE-OPT_­stefan-boltzmann_­example_­01_­MPI_­4	Passed	Completed	19	0	19	#8613
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­test_­sol_­checkSuperQuantileQuadrangle_­MPI_­1	Passed	Completed	19	0	19	#8613

Tests with issue trackers Failed: twif=4

Site	Build Name	Test Name	Status	Details	Consec­utive Non-pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	18	18	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	28	28	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	24	24	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	24	24	0	#8613

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[ccober6]

@dridzal Do you have a timeframe when to can look at this?

[grover-trilinos]

Test results for issue #8613 as of 2021-06-06

Tests with issue trackers Passed: twip=2
Tests with issue trackers Failed: twif=4

Detailed test results: (click to expand)

Tests with issue trackers Passed: twip=2

Site	Build Name	Test Name	Status	Details	Consec­utive Pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­example_­PDE-OPT_­stefan-boltzmann_­example_­01_­MPI_­4	Passed	Completed	18	0	18	#8613
vortex	Trilinos-atdm-ats2-gnu-7.3.1-spmpi-rolling_­serial_­static_­dbg	ROL_­test_­sol_­checkSuperQuantileQuadrangle_­MPI_­1	Passed	Completed	18	0	18	#8613

Tests with issue trackers Failed: twif=4

Site	Build Name	Test Name	Status	Details	Consec­utive Non-pass Days	Non-pass Last 30 Days	Pass Last 30 Days	Issue Tracker
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	22	22	0	#8613
eclipse	Trilinos-atdm-cts1-intel-19.0.4_­openmpi-4.0.3_­openmp_­static_­opt	ROL_­example_­tempus_­example_­02_­MPI_­1	Failed	Completed (Failed)	28	28	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.0_­openmpi-4.0.2_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	24	24	0	#8613
stria	Trilinos-atdm-van1-tx2_­arm-20.1_­openmpi-4.0.3_­openmp_­static_­dbg	ROL_­test_­vector_­StdArrayInterface_­MPI_­1	Failed	Completed (Failed)	24	24	0	#8613

This is an automated comment generated by Grover. Each week, Grover collates and reports data from CDash in an automated way to make it easier for developers to stay on top of their issues. Grover saw that there are tests being tracked on CDash that are associated with this open issue. If you have a question, please reach out to Ross. I'm just a cat.

[ccober6]

@dridzal Sorry to bother you with this, but we are trying to clean things. Two tests are passing, but it looks like two test are consistently failing on eclipse and stria. Thanks, Curt.

[dridzal]

@ccober6 One of the failing tests is a total mystery, but we'll look into it. For the other test,

https://testing-dev.sandia.gov/cdash/test/53688081

I may need some help from @bartlettroscoe : The failure is due to hanging RCP nodes. In this example we create a memory pool using std::array, which is allocated on the stack, to hold RCPs to vectors, which are allocated dynamically later. @gregvw and I have had discussions about whether this could lead to issues in the deallocation, as std::array may have mechanisms that conflict with RCP deallocation. Do you see any issues with this? We could use std::vector instead of std::array, of course, if needed.

[bartlettroscoe]

The failure is due to hanging RCP nodes

I think the RCP printout at the end is just a symptom of the test segfalting and not calling destructors properly on objects in main(). If you look at the bottom of

https://testing-dev.sandia.gov/cdash/test/53688081

you see:

[st2:93678] *** Process received signal ***
[st2:93678] Signal: Aborted (6)
[st2:93678] Signal code:  (-6)

When a program segfaults that uses Teuchos::RCP, it seems that destructors on static objects are often called. That is where these hanging RCP references come from.

What happens if you run this test with valgrind on this system or other systems? What about address sanitizer? (I wish Trilinos has regular dynamic testing set up to make this sort of thing easier.)

[dridzal]

Thanks @bartlettroscoe . I'll run valgrind on it. However, if you look further up, the test reports "TEST PASSED", which is the last operation before destructors are called (automatically). After this operation, I'm pretty sure that the program goes into deallocating std::array<RCP>.

[bartlettroscoe]

However, if you look further up, the test reports "TEST PASSED"

@dridzal, unfortunately, it is pretty common for a test to compute the expected result but have a memory usage problem that results in a segfault as the program is shutting down. In fact, this happens so frequently that I added a special check for this with tribits_add_advanced_test() with an option called ALWAYS_FAIL_ON_NONZERO_RETURN so you can require that STDOUT provide a given string (like "TEST PASSED") AND return a zero return code (which test did not as it returned a value of 134). (Raw ctest has no such option.) C++ is so much fun.