hydromt icon indicating copy to clipboard operation
hydromt copied to clipboard

Change the sequence of renaming and the unit_add/unit_mult/astype preprocessing steps in data adapter

Open hboisgon opened this issue 3 years ago • 1 comments

Currently in the data_adapter, renaming of variables happens before unit conversion or type checks.

It would be more consistent to first do:

  • astype
  • unit_add
  • unit_mult
  • renmaing last

This has consequences on the data catalog format. The dictionnary would then always have the variables names as in the original data as keys (instead of already the renamed variable for unit_mult and unit_add). This impacts the deltares_data.yml file stored in data-artifacts but not the data extract for Piave itself.

hboisgon avatar Jul 23 '21 14:07 hboisgon

The current sequence (first renaming) is because the .raster and .vector extensions for Datasets/Arrays require correct x, y, time and index dimension names. Some minimal renaming of dimensions should therefore be the first step after reading. Note that nodata values can also be set per variable. My suggested order of the pre-processing would therefore be:

  • renaming dimensions
  • nodata
  • astype
  • unit_add
  • unit_mult
  • renaming variables

DirkEilander avatar Jul 26 '21 10:07 DirkEilander

This has been fixed in #481

DirkEilander avatar Oct 18 '23 13:10 DirkEilander