disdrodb icon indicating copy to clipboard operation
disdrodb copied to clipboard

Allow integer type for L0A parquet files

Open ghiggi opened this issue 2 years ago • 0 comments

Description

In this commit I had to remove all integer types from the L0A_encodings.yml files because if there is a nan value in an integer column, an error will be raised during column casting.

This change increases the size of the L0A Parquet files.

To reintroduce the integer type, it would be required:

  • to define a dictionary with the nan flag value for each variable which is expected to be cast to an integer format.
  • during L0A processing replace the nan values using the nan_flag dictionary
  • during L0B processing, replace the nan_flag with np.nan (by casting the variable to float) or ensure that the nan_flag value corresponds to the _FillValue of the L0B encoding.

ghiggi avatar Nov 11 '22 13:11 ghiggi