ESCOMP
"nn" mapalgo option for streams can be inconsistent with pelayout and be incorrect
We found this in #3634 for Carbon Isotopes. But, this is a general problem for using the "nn" mapalgo option for streams for ESMF versions before ESMF-8.9.0.
OK, the failing test does show up the same on: derecho_gnu and derecho_intel. It doesn't fail for an ERS test. This means it's due to the change in processor count on restart.
The difference is a single point in Africa near the latitude band where the nn (Nearest Neighbor) interpolation picks a different point when run with a different number of processes. In one case in the more Northern band and in the other in the more Southern band.
I looked at the half degree grid to make sure there wasn't weirdness there -- and there isn't. So it's something in the nn interpolation in streams. Here's some plots that show the issue. And looking at it there are some other discrepancies over the latitude band that aren't right either since the fv10x15 grid has constant latitudes across the globe.
Originally posted by @ekluzek in https://github.com/ESCOMP/CTSM/issues/3634#issuecomment-3633975474
More comments from the PR:
Code in CDEPS where this comes into play:
https://github.com/ESCOMP/CTSM/pull/3634#issuecomment-3633993717
Testing with bilinear the desintation grid option for NN and using ESMF-8.9.0:
https://github.com/ESCOMP/CTSM/pull/3634#issuecomment-3635600688
Note that this same test passes for example on the ne30 grid or slightly different PE counts.
3 48x1 Izumi tests: ERP_D_Ld5_P48x1.f10_f10_mg37.I1850Clm50Bgc.izumi_nag.clm-ciso ERP_D_Ld5_P48x1.f10_f10_mg37.I1850Clm60Bgc.izumi_nag.clm-ciso ERP_D_Ld5_P48x1.f10_f10_mg37.I1850Clm60Bgc.izumi_nag.clm-ciso--clm-matrixcnOn
5 64x2 Derecho tests: ERP_D_Ld10_P64x2.f10_f10_mg37.IHistClm60BgcCrop.derecho_intel.clm-ciso_decStart ERP_D_Ld10_P64x2.f10_f10_mg37.IHistClm60BgcCrop.derecho_intel.clm-ciso_decStart--clm-matrixcnOn_ignore_warnings ERP_P64x2_D_Ld5.f10_f10_mg37.I1850Clm45BgcCru.derecho_intel.clm-ciso ERP_P64x2_D_Ld5.f10_f10_mg37.I1850Clm50Bgc.derecho_intel.clm-ciso ERP_P64x2_D_Ld5.f10_f10_mg37.I1850Clm50Bgc.derecho_intel.clm-ciso--clm-matrixcnOn_ignore_warnings
2 f10 Derecho tests: ERP_D_Ld5.f10_f10_mg37.I2000Clm50BgcCru.derecho_gnu.clm-ciso_flexCN_FUN ERP_D_Ld5.f10_f10_mg37.I2000Clm50BgcCru.derecho_gnu.clm-ciso_flexCN_FUN--clm-matrixcnOn
2 ne30pg3 Derecho tests: ERP_D_Ld9.ne30pg3_t232.I1850Clm60BgcCropG.derecho_intel.clm-clm60cam7LndTuningModeLDust ERP_D_Ld9.ne30pg3_t232.IHistClm60BgcCropG.derecho_intel.clm-clm60cam7LndTuningModeLDust
1 fv45 Derecho test: ERP_Ld9.f45_g37.I2000Clm60BgcCrujra.derecho_intel.clm-default
So I think this has to happen for a grid that has a latitude line for a change that exactly matches the model grid. So f10 has it, but ne30pg3 doesn't.