[bug from Hell] Serious issue with: concatenated cubes with masks and `dask=2024.8.0`

[valeriupredoi]

Hi folks,

• MRE/MRC code:

import iris
import numpy as np


c1 = iris.load_cube("cubb-1.nc")
c2 = iris.load_cube("cubb-2.nc")

# apply slice to concatenated cube
slicer = (
    np.random.choice(a=[False, True], size=(730,)),
    slice(None, None, None),
    slice(None, None, None),
    slice(None, None, None)
)

# can use this to slice each of cubes c1 and/or c2
slicer1 = (
    np.random.choice(a=[False, True], size=(365,)),
    slice(None, None, None),
    slice(None, None, None),
    slice(None, None, None)
)

cube = iris.cube.CubeList([c1, c2]).concatenate_cube()
cubes = cube[slicer]
print("After slicing", cubes.data)

• environment:

iris                      3.9.0              pyha770c72_0    conda-forge
numpy                     1.26.4          py312heda63a1_0    conda-forge

dask                      2024.8.0           pyhd8ed1ab_0    conda-forge
dask-core                 2024.8.0           pyhd8ed1ab_0    conda-forge

or

dask                      2024.7.1           pyhd8ed1ab_0    conda-forge
dask-core                 2024.7.1           pyhd8ed1ab_0    conda-forge

• problem: use the MRE code to reproduce this:
  • you have two cubes, one or both with masked data
  • concatenate them
  • apply a (time-like) slice
  • resulting cube will randomly (depending on slice) have fill values explicitly used as numerical data values (ie not as masked elements) when using dask==2024.8.0, so one gets cray values of 1.e+36 etc
  • this behaviour never happens with previous dask==2024.7.1
  • example above uses fairly hefty cubes (attached here for testing, just change the extension), but am sure this can be scaled-down to smaller cubes, with the same behaviour

Good luck fixing this folks, it took me two days to isolate it from ESMValCore, am sure it's not a very straightforward fix :grin: But it's an ugly bug that can bite badly! cubb-1.nc.txt cubb-2.nc.txt