FAIL in aux_cam_noresm test: ERP_Ln9.ne16pg3_ne16pg3_mtn14.NF1850ghg_fates-nocomp.betzy_intel.cam-outfrq9s

[TomasTorsvik]

Describe the bug Please provide a clear and concise description of what the bug is.

• NorESM version: noresm3_0_alpha06
• HPC platform: bertzy
• Compiler (if applicable): intel
• Compset (if applicable): NF1850ghg_fates-nocomp
• Resolution (if applicable): ne16pg3_ne16pg3_mtn14
• Error message (if applicable): FAIL on COMPARE_base_rest test

    ERP_Ln9.ne16pg3_ne16pg3_mtn14.NF1850ghg_fates-nocomp.betzy_intel.cam-outfrq9s.GC.20250620_124958_esb3vy.cpl.hi.0001-01-01-16200.nc.base matched ERP_Ln9.ne16pg3_ne16pg3_mtn14.NF1850ghg_fat
es-nocomp.betzy_intel.cam-outfrq9s.GC.20250620_124958_esb3vy.cpl.hi.0001-01-01-16200.nc.rest
FAIL
 ---------------------------------------------------
2025-06-20 13:31:57: compared suffixes suffix1 'base' suffix2 'rest'

tail -n20 /cluster/work/users/tomast/noresm/nor3_a07/ERP_Ln9.ne16pg3_ne16pg3_mtn14.NF1850ghg_fates-nocomp.betzy_intel.cam-outfrq9s.GC.20250620_124958_esb3vy/run/ERP_Ln9.ne16pg3_ne16pg3_mtn14.
NF1850ghg_fates-nocomp.betzy_intel.cam-outfrq9s.GC.20250620_124958_esb3vy.clm2.h0.0001-01-01-16200.nc.base.cprnc.out

                9864   0.812818054321473E+00   0.444444444444444E-01
                9864   0.812818054321473E+00   0.444444444444444E-01
               27648  ( 10748,     1,     1) ( 12296,     1,     1)
          avg abs field values:    1.391095850044782E-01
                                   1.391095850044782E-01
************************************************************************************************************************************
  
SUMMARY of cprnc:
 A total number of    602 fields were compared
          of which     10 had non-zero differences
               and      0 had differences in fill patterns
               and      0 had different dimension sizes
               and      0 had different data types
 A total number of      2 fields could not be analyzed
 A total number of      0 time-varying fields on file 1 were not found on file 2.
 A total number of      0 time-constant fields on file 1 were not found on file 2.
 A total number of      0 time-varying fields on file 2 were not found on file 1.
 A total number of      0 time-constant fields on file 2 were not found on file 1.
  diff_test: the two files seem to be DIFFERENT 

10 fields differ between base and rest:

• FATES_CBALANCE_ERROR
• FATES_MORTALITY_CFLUX_USTORY
• FATES_SEEDS_IN
• FATES_ERROR_EL
• FATES_SEEDS_IN_EXTERN_EL
• FATES_MORTALITY_CFLUX_PF
• FATES_MORTALITY_PF
• FATES_MORTALITY_IMPACT_SZ
• FATES_MORTALITY_USTORY_SZ
• FATES_FRAGMENTATION_SCALER_SL

To Reproduce Steps to reproduce the behavior:

1. check out noesm3_0_alpha06
2. run aux_cam_noresm test suite

Expected behavior COMPARE_base_rest test should PASS

Additional context This appears to be a problem specific to NF1850ghg + fates-nocomp, possibly for this particular grid. A similar test case with compset NF1850ghg + fates-sp does not fail on COMPARE_base_rest

[maritsandstad]

I've run aux_cam_noresm with this and new CTSM-tag: ctsm5.3.045_noresm_v8 on top for ctsm NBP fluxes, I get the same exact fail.

[rosiealice]

FATES-SP doesn't use any of those fields so wouldn't produce this error. Does this test pass with previous tags?

[TomasTorsvik]

@rosiealice , @maritsandstad The NF1850ghg is a new capability introduced in one of the latest CAM tags, so it is not available for earlier tags. I talked with Dirk earlier today, he had been testing this with the ne30pg3 grid, but for some reason the test was created for the ne16pg3 grid. I guess something has not been set properly for this specific compset and grid configuration.

[mvdebolskiy]

Ln9 should be changed to Ld*.

[mvertens]

@mvdebolskiy - most cam tests use 9 time steps and don't run for multiple days. Does FATES not have the capability to restart after 5 time steps? @gold2718 - do CAM tests currently test FATES? This would be good to do given that it is the default for NorESM3.

[mvdebolskiy]

@mvertens no, there are some variables that do not have a proper restart for restarting within a day. And it's some work to find and fix all of them, so it's not a priority.