Merge dimension mashup

[bblay]

🐛 Bug Report

Merge is mashing two dimensions into one.

How To Reproduce

I didn't manage to reproduce this using contrived cubes, so I can only currently reproduce it from the source data, at work. Simply load the cubes and the problem occurs. I've written a script which demonstrates the problem and will paste that, and the output, below. I also have a workaround at work. I'm not yet sure if I can share it.

Expected behaviour

The data are global lat-lon octants. I believe they should be loaded into sixteen cubes. Eight of them should share a single time and forecast period, with fourteen pressure levels. The other eight should share a different time and forecast period, and have 11 pressure levels.

Actual behaviour

Instead, the data is loading into eight cubes with the third dimension having 25 points - a mashup of the pressure and time coords.

One impact of this: I'm extracting each time point and saving as netcdf. Without the workaround, it produces a file without proper knowledge of the pressure dimension. It's just called "dim1", it's in a seemingly random order, and the points are just indices 0-13. Another impact is that the coord points are a bit of a mess, as you can see in the output below.

Workaround

I've worked around this by using load_raw, and writing a function which groups by time, merging each group separately to force the pressure levels to be merged into a sensible dimension first. The workaround is specific to the data, not a generic fix.

Environment

• OS & Version: RHEL 7.9
• Iris Version: 3.7

Additional context

code to highlight the problem

#!/usr/bin/env python

from pathlib import Path
import sys
sys.path.append('HIDING_PATHS_HERE')

import iris
from iris.cube import CubeList

# workaround is in here
from grib_concat import load_cubes, load_cubes_controlled


if __name__ == '__main__':

    fpaths = [
        Path('HIDING_PATHS_HERE/file1'),
        Path('HIDING_PATHS_HERE/file2'),
    ]

    # normal iris loading
    iris_load_cubes = load_cubes(fpaths)
    iris_156_cubes = CubeList([c for c in iris_load_cubes if 'UNKNOWN LOCAL PARAM 156' in c.name()])

    # workaround with controlled merging
    controlled_merge_cubes = load_cubes_controlled(fpaths)
    controlled_156_cubes = CubeList([c for c in controlled_merge_cubes if 'UNKNOWN LOCAL PARAM 156' in c.name()])

    print('--- iris load ---\n')

    print(f'{len(iris_156_cubes)} cubes\n')
    for c in iris_156_cubes:
        print(repr(c))

    print('\ncube 0:\n', iris_156_cubes[0], '\n')
    print('forecast_period', iris_156_cubes[0].coord('forecast_period').points)
    print('pressure', iris_156_cubes[0].coord('pressure').points)
    print('time', iris_156_cubes[0].coord('time').points, '\n')

    print('one of the cubes after extracting fp=0:')
    iris_fp0 = iris_156_cubes.extract(iris.Constraint(forecast_period=0))[0]
    print(repr(iris_fp0))
    print('forecast_period', iris_fp0.coord('forecast_period').points)
    print('pressure', iris_fp0.coord('pressure').points)
    print('time', iris_fp0.coord('time').points, '\n')
    iris.save(iris_fp0, 'from_iris_load.nc')

    print('--- controlled merge ---\n')

    print(f'{len(controlled_156_cubes)} cubes\n')
    for c in controlled_156_cubes:
        print(repr(c))

    print('\ncube 0:\n', controlled_156_cubes[0], '\n')
    print('forecast_period', controlled_156_cubes[0].coord('forecast_period').points)
    print('pressure', controlled_156_cubes[0].coord('pressure').points)
    print('time', controlled_156_cubes[0].coord('time').points, '\n')

    print('one of the cubes after extracting fp=0:')
    controlled_fp0 = controlled_156_cubes.extract(iris.Constraint(forecast_period=0))[0]
    print(repr(controlled_fp0))
    print('forecast_period', controlled_fp0.coord('forecast_period').points)
    print('pressure', controlled_fp0.coord('pressure').points)
    print('time', controlled_fp0.coord('time').points, '\n')
    iris.save(controlled_fp0, 'from_controlled_merge.nc')

In the output below, you can first see the data loaded with iris.load, then using iris.load_raw and "controlled" merging. In both cases one cube is printed, and then cubes are extracted at forecast_period=0 with one of those printed.

output

--- iris load ---

8 cubes

<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (-- : 25; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (-- : 25; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (-- : 25; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (-- : 25; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (-- : 25; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (-- : 25; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (-- : 25; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (-- : 25; latitude: 61; longitude: 60)>

cube 0:
 UNKNOWN LOCAL PARAM 156.128 / (unknown) (-- : 25; latitude: 61; longitude: 60)
    Dimension coordinates:
        latitude                            -             x              -
        longitude                           -             -              x
    Auxiliary coordinates:
        forecast_period                     x             -              -
        pressure                            x             -              -
        time                                x             -              -
    Scalar coordinates:
        originating_centre              European Centre for Medium Range Weather Forecasts
    Attributes:
        generatingProcessIdentifier     154
        indicatorOfParameter            156
        indicatorOfTypeOfLevel          'pl'
        originating_centre_id           'ecmf' 

forecast_period [ 0  0  0  0  0  0  0  0  0  0  0  0  0  0 24 24 24 24 24 24 24 24 24 24
 24]
pressure [ 300  250  400 1000  700  925  200  850  500  100  150   50   70   30
  700  925  300  200  850 1000  100  400  500  250  150]
time [474240. 474240. 474240. 474240. 474240. 474240. 474240. 474240. 474240.
 474240. 474240. 474240. 474240. 474240. 474264. 474264. 474264. 474264.
 474264. 474264. 474264. 474264. 474264. 474264. 474264.] 

one of the cubes after extracting fp=0:
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (-- : 14; latitude: 61; longitude: 60)>
forecast_period [0 0 0 0 0 0 0 0 0 0 0 0 0 0]
pressure [ 300  250  400 1000  700  925  200  850  500  100  150   50   70   30]
time [474240. 474240. 474240. 474240. 474240. 474240. 474240. 474240. 474240.
 474240. 474240. 474240. 474240. 474240.] 

--- controlled merge ---

16 cubes

<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 14; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 14; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 14; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 14; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 14; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 14; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 14; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 14; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 11; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 11; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 11; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 11; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 11; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 11; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 11; latitude: 61; longitude: 60)>
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 11; latitude: 61; longitude: 60)>

cube 0:
 UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 14; latitude: 61; longitude: 60)
    Dimension coordinates:
        pressure                                 x             -              -
        latitude                                 -             x              -
        longitude                                -             -              x
    Scalar coordinates:
        forecast_period                 0 hours
        originating_centre              European Centre for Medium Range Weather Forecasts
        time                            2024-02-07 00:00:00
    Attributes:
        generatingProcessIdentifier     154
        indicatorOfParameter            156
        indicatorOfTypeOfLevel          'pl'
        originating_centre_id           'ecmf' 

forecast_period [0]
pressure [  30   50   70  100  150  200  250  300  400  500  700  850  925 1000]
time [474240.] 

one of the cubes after extracting fp=0:
<iris 'Cube' of UNKNOWN LOCAL PARAM 156.128 / (unknown) (pressure: 14; latitude: 61; longitude: 60)>
forecast_period [0]
pressure [  30   50   70  100  150  200  250  300  400  500  700  850  925 1000]
time [474240.]