Andrew M. Bradley

I've isolated at least one nondeterministic diff to P3 (small-kernels version). I'll first try the deopt that worked for the (differently expressed) issue with P3 using ROCm 5.7. I'll also...

@trey-ornl the P3 deoptimization is working. Thus, in your performance testing, if you haven't started already, I suggest adding this: ```diff diff --git a/cime_config/machines/cmake_macros/craygnu-hipcc.cmake b/cime_config/machines/cmake_macros/craygnu-hipcc. cmake index 8322c5d3a9..6cb79c0146 100644 ---...

I don't use those tests for nondeterminism analysis and have not run them in years. I consider a single-node ne30 ERS test to be the most useful test configuration.

This might be specific to Intel. We have [PEM_Ln90.ne30pg2_ne30pg2.F2010-SCREAMv1.pm-cpu_gnu.scream-spa_remap--scream-output-preset-4](https://my.cdash.org/test/209360528) in our nightly, which is GNU on pm-cpu.

@oksanaguba should we support both column and global integrals? It seems to me we should.

Re: ERS_Ld5.T62_oQU240wLI.GMPAS-DIB-IAF-DISMF.pm-cpu_intel, does the MPAS seaice log show anything? There might also be MPAS error files that give details. I base this guess on the stack trace you posted.

> How it possible to be non-bfb at one point and then "recover" and be BFB the remaining steps? These are the only diffs in the file after 18 steps....

Orthogonal comment (i.e., neutral w.r.t. the specific issue described above): One way to bring in a different Kokkos, including for standalone Homme and with the benefit of some level of...

Ok, sounds good. I know some of the syncs were added in response to nondeterminism, but that was years ago and Kokkos has changed.

@singhbalwinder is it possible that the issue is that in, e.g., this line: ```F90 qv_dry = calculate_drymmr_from_wetmmr(ncol, pver, qv_wet_in, qv_wet_in) ``` `qv_dry` is declared as `qv_dry(pcols,pver)` but `calculate_drymmr_from_wetmmr` returns an...