Track global indices when using a subset with chunked-rwg

[bekozi]

When a spatial subset is used with a chunked regridding operation, the mapping to global indices is lost. These indices should be tracked and used during weight file reconstruction.

[ekluzek]

Note, this issue is important for CTSM... https://github.com/ESCOMP/ctsm/issues/806

[bekozi]

Starting to look into this and plan to collect requirements here:

• [ ] Concurrent generation of ESMF equivalent global indices prior to subsetting the source field.
  • Indices need to be added to the source field as a secondary variable accessible during the weight file merging and remapping.
  • Current implementation in ocgis only works in serial as part of the chunked weight file merging.
  • Needs to work with unstructured data but is an easier problem since the identifiers are 1D.
• [x] ESMF equivalent local indices added to the subset file to provide the second mapping component alongside the subset global indices.
• [x] Source global to source subset weight index remapping occurs during merge/format.
  • The weight file merging method should be renamed to indicate it is a formatting step that may or may not be merg. The method can also remap weight files associated with a single spatial subset file and no chunked weight files.
  • The source local/subset indices + the global indices are used to remap the weight file so it may be used during an SMM using the global (not subsetted) source file.
• [ ] Allow control of index remapping at the CLI level.
  • Option can work with and without weight file merge and could be called src_index_remap.
  • When on, the option will look for the index variables added during the subset process.
• [ ] Get the remap to work in parallel.
  • The reduction used on coordinates indices in UGRID can probably be reused for the remapping.
  • The operation can probably happen pretty fast in serial since it is just searching integer arrays.

[bekozi]

• Started this work in https://github.com/NCPP/ocgis/tree/i494-track-indices.
• The related esmpy branch is: https://github.com/esmf-org/esmf/tree/ESMPy-arbindices

The approach no longer needs the remapping in ocgis since we can set the arbitrary sequence indices directly on the esmf grid through python. Ocgis will just detect the indices on the source grid and use those when calling esmf weight generation.

[bekozi]

Arbitrary sequence indices are only implemented for Grid objects in ESMPy. The Mesh global sequence index tracking will take some tweaking to ESMF IO routines. For grids, the global index tracking is working in parallel. I'll update with Mesh progress as it moves forward.

[bekozi]

We finally identified the right approach for this that is sustainable moving forward. When the MOAB backend is enabled for the ESMF Mesh (required for ESMF version 8.1 forward), we can modify the sequence indices of Mesh after it is created. This way we don't have to mess around with quasi-IO standards when modifying the indexing used by the regridding for meshes. @rokuingh is working on changing the mesh indices using ESMF-MOAB.