YAXArrays.jl icon indicating copy to clipboard operation
YAXArrays.jl copied to clipboard

Should cubes keep the missing_value and _FillValue when changing the eltype?

Open felixcremer opened this issue 2 years ago • 1 comments

I am constructing a mask from an existing cube by mapping a function which returns a boolean. This results in a cube with eltype Union{Missing, Bool}. Since this cube comes from a zarr datacube it copies the _FillValue and the missing_value in the properties from the source data cube. This leads to errors down the line when saving the data cube to disk, because we get an inexact error, because the _FillValue can't be converted to a Boolean.

We can circumvent the problem by assigning new values to the missingvalue and _FillValue entries in cube.properties but should we have a more automatic approach in the map function?

julia> signifmask = map(!iszero, wplabels)
YAXArray with the following dimensions
X                   Axis with 7500 Elements from 9.48724066e6 to 9.26227066e6
Y                   Axis with 5820 Elements from 561609.15 to 736179.15
Polarisation        Axis with 2 elements: VH VV 
name: layer
Total size: 83.26 MB


julia> signifmask.properties
Dict{String, Any} with 3 entries:
  "missing_value" => 9223372036854775807
  "name"          => "layer"
  "_FillValue"    => 9223372036854775807

julia> wplabels.properties
Dict{String, Any} with 3 entries:
  "missing_value" => 9223372036854775807
  "name"          => "layer"
  "_FillValue"    => 9223372036854775807

julia> eltype(signifmask)
Union{Missing, Bool}

julia> eltype(wplabels)
Union{Missing, Int64}

felixcremer avatar Feb 10 '23 13:02 felixcremer

Very good question. In general I think it would be better to not carry the missing_value and fillvalue into the YAXArray object at all, because they should be obsolete once we have applied the CF conventions. Maybe this is worth doing.

meggart avatar Apr 18 '23 12:04 meggart