Error! Failed post-condition property check in p3 for one member of autotune cases at ne1024

[ndkeen]

Found this error for a ne1024 case on frontier. Ran it again with same result. This is one of the autotune cases, where we restart from a given set of files, make some changes to P3/SHOC params, and run 2 days. We were not expecting all parameter combinations to work, but out of over 160 cases that we have tried so far, this is the only actual fail.

3430: WARNING: Tl1_1 has 1 values <= allowable value.  Resetting to minimum value.
   0: bfbhash>           5544 a5782b16cbde238b (Hommexx)
   0: bfbhash>           5562 b7e2d2a9f555bdb7 (Hommexx)

3771:  FAIL:
3771: false
3771: /lustre/orion/cli115/proj-shared/noel/wacmy/s10-feb7/components/eamxx/src/share/atm_process/atmosphere_process.cpp:454
3771: Error! Failed post-condition property check (cannot be repaired).
3771:   - Atmosphere process name: p3
3771:   - Property check name: qr within interval [0, 0.1]
3771:   - Atmosphere process MPI Rank: 3771
3771:   - Message: Check failed.
3771:   - check name: qr within interval [0, 0.1]
3771:   - field id: qr[Physics PG2] <double:ncol,lev>(6144,128) [kg/kg]
3771:   - minimum:
3771:     - value: 0
3771:     - indices (w/ global column index): (10750785,0)
3771:     - lat/lon: (5.36299, 198.303)
3771:     - additional data (w/ local column index):
3771:
3771:      phis<ncol>(6144)
3771:
3771:   phis(0)
3771:     0,
3771:
3771:      landfrac<ncol>(6144)
3771:
3771:   landfrac(0)
3771:     0,
3771:
3771:     END OF ADDITIONAL DATA
3771:
3771:   - maximum:
3771:     - value: 0.104302
3771:     - indices (w/ global column index): (10804185,88)
3771:     - lat/lon: (6.3149, 201.643)
3771:     - additional data (w/ local column index):
3771:
3771:      phis<ncol>(6144)
3771:
3771:   phis(2648)
3771:     0,

/lustre/orion/cli115/proj-shared/noel/e3sm_scratch/s10-feb7/dd1024/m0107/SCREAM.2024-autocal-00.ne1024pg2

    # Parameters (workdir nlhs=  107) from /lustre/orion/cli115/proj-shared/wagmanbe/dakota/eamxx_olcf/production/ens
    ./atmchange physics::mac_aero_mic::shoc::thl2tune=1.630303079000000e+00
    ./atmchange physics::mac_aero_mic::shoc::qw2tune=3.579987221000000e+00
    ./atmchange physics::mac_aero_mic::shoc::length_fac=4.387730082000000e+00
    ./atmchange physics::mac_aero_mic::shoc::c_diag_3rd_mom=5.422601655000000e+00
    ./atmchange physics::mac_aero_mic::shoc::Ckh=6.331664519000000e-01
    ./atmchange physics::mac_aero_mic::shoc::Ckm=3.315688891000000e-01
    ./atmchange physics::mac_aero_mic::shoc::lambda_low=3.414475691000000e-03
    ./atmchange physics::mac_aero_mic::shoc::lambda_high=9.938565966000000e-02
    ./atmchange physics::mac_aero_mic::p3::p3_spa_to_nc=9.745559715000000e-01
    ./atmchange physics::mac_aero_mic::p3::p3_eci=1.438633836000000e-01
    ./atmchange physics::mac_aero_mic::p3::p3_eri=9.761191496000000e-01
    ./atmchange physics::mac_aero_mic::p3::p3_k_accretion=2.452719371000000e+00
    ./atmchange physics::mac_aero_mic::p3::p3_dep_nucleation_exponent=2.557150141000000e-01
    ./atmchange physics::mac_aero_mic::p3::max_total_ni=9.290880092000000e+06
    ./atmchange physics::mac_aero_mic::p3::p3_ice_sed_knob=8.539005624000000e-01
    ./atmchange physics::mac_aero_mic::p3::p3_d_breakup_cutoff=5.509715800000000e-06