disdrodb
disdrodb copied to clipboard
Allow integer type for L0A parquet files
Description
In this commit I had to remove all integer types from the L0A_encodings.yml files because if there is a nan value in an integer column, an error will be raised during column casting.
This change increases the size of the L0A Parquet files.
To reintroduce the integer type, it would be required:
- to define a dictionary with the nan flag value for each variable which is expected to be cast to an integer format.
- during L0A processing replace the nan values using the nan_flag dictionary
- during L0B processing, replace the nan_flag with np.nan (by casting the variable to float) or ensure that the nan_flag value corresponds to the _FillValue of the L0B encoding.