SciTools
Iris won't load netCDF files from the new CDS-beta
🐛 Bug Report
Iris won't load netCDF files from the new CDS-beta. Someone has also reported this on a post about the updates to the CDS API here (https://forum.ecmwf.int/t/changes-to-grib-to-netcdf-converter-on-cds-beta-ads-beta/4322/19). I suspect the issue comes from the changes to their converter described there. However, I think iris not loading the files is an iris bug. If I change the offending line (traceback below) in iris from
total_bytes = cf_var.size * cf_var.dtype.itemsize
to
total_bytes = cf_var.size * np.dtype(cf_var.dtype).itemsize
Then the file loads fine. Simple fix but I thought it's worth reporting in case this might lead to other issues elsewhere.
Running ncdump on the file shows up the difference too. The variables look like this. The string in front of the string variables wouldn't have been there before. I don't know if that is an issue or not.
double latitude(latitude) ;
latitude:_FillValue = NaN ;
string latitude:units = "degrees_north" ;
string latitude:standard_name = "latitude" ;
string latitude:long_name = "latitude" ;
string latitude:stored_direction = "decreasing" ;
How To Reproduce
Steps to reproduce the behaviour:
- Download a netCDF file from CDS-beta. I just ticked the first box for each value here (https://cds-beta.climate.copernicus.eu/datasets/reanalysis-era5-pressure-levels?tab=download), i.e. Divergence at 1hPa, 1st January 1940
- Try to open it with iris
Expected behaviour
The netCDF file should load. It works with xarray and CF python
Additional context
Click to expand this section...
AttributeError Traceback (most recent call last) Cell In[2], line 1 ----> 1 cubes = iris.load("test_era5.nc")
File ~/miniforge3/envs/core/lib/python3.12/site-packages/iris/init.py:326, in load(uris, constraints, callback) 302 def load(uris, constraints=None, callback=None): 303 """Load any number of Cubes for each constraint. 304 305 For a full description of the arguments, please see the module (...) 324 325 """ --> 326 return _load_collection(uris, constraints, callback).merged().cubes()
File ~/miniforge3/envs/core/lib/python3.12/site-packages/iris/init.py:294, in _load_collection(uris, constraints, callback) 292 try: 293 cubes = _generate_cubes(uris, callback, constraints) --> 294 result = _CubeFilterCollection.from_cubes(cubes, constraints) 295 except EOFError as e: 296 raise iris.exceptions.TranslationError( 297 "The file appears empty or incomplete: {!r}".format(str(e)) 298 )
File ~/miniforge3/envs/core/lib/python3.12/site-packages/iris/cube.py:97, in _CubeFilterCollection.from_cubes(cubes, constraints) 95 pairs = [_CubeFilter(constraint) for constraint in constraints] 96 collection = _CubeFilterCollection(pairs) ---> 97 for cube in cubes: 98 collection.add_cube(cube) 99 return collection
File ~/miniforge3/envs/core/lib/python3.12/site-packages/iris/init.py:275, in _generate_cubes(uris, callback, constraints) 273 if scheme == "file": 274 part_names = [x[1] for x in groups] --> 275 for cube in iris.io.load_files(part_names, callback, constraints): 276 yield cube 277 elif scheme in ["http", "https"]:
File ~/miniforge3/envs/core/lib/python3.12/site-packages/iris/io/init.py:219, in load_files(filenames, callback, constraints) 217 fnames = handler_map[handling_format_spec] 218 if handling_format_spec.constraint_aware_handler: --> 219 for cube in handling_format_spec.handler(fnames, callback, constraints): 220 yield cube 221 else:
File ~/miniforge3/envs/core/lib/python3.12/site-packages/iris/fileformats/netcdf/loader.py:641, in load_cubes(file_sources, callback, constraints) 638 if mesh is not None: 639 mesh_coords, mesh_dim = _build_mesh_coords(mesh, cf_var) --> 641 cube = _load_cube(engine, cf, cf_var, cf.filename) 643 # Attach the mesh (if present) to the cube. 644 for mesh_coord in mesh_coords:
File ~/miniforge3/envs/core/lib/python3.12/site-packages/iris/fileformats/netcdf/loader.py:326, in _load_cube(engine, cf, cf_var, filename) 324 these_settings = CHUNK_CONTROL.var_dim_chunksizes.get(cf_var.cf_name, {}) 325 with CHUNK_CONTROL.set(**these_settings): --> 326 return _load_cube_inner(engine, cf, cf_var, filename)
File ~/miniforge3/envs/core/lib/python3.12/site-packages/iris/fileformats/netcdf/loader.py:355, in _load_cube_inner(engine, cf, cf_var, filename) 350 _assert_case_specific_facts(engine, cf, cf_var.cf_group) 352 # Run the actions engine. 353 # This creates various cube elements and attaches them to the cube. 354 # It also records various other info on the engine, to be processed later. --> 355 engine.activate() 357 # Having run the rules, now add the "unused" attributes to each cf element. 358 def fix_attributes_all_elements(role_name):
File ~/miniforge3/envs/core/lib/python3.12/site-packages/iris/fileformats/_nc_load_rules/engine.py:95, in Engine.activate(self) 85 def activate(self): 86 """Run all the translation rules to produce a single output cube. 87 88 This implicitly references the output variable for this operation, (...) 93 94 """ ---> 95 run_actions(self)
File ~/miniforge3/envs/core/lib/python3.12/site-packages/iris/fileformats/_nc_load_rules/actions.py:575, in run_actions(engine) 573 auxcoord_facts = engine.fact_list("auxiliary_coordinate") 574 for auxcoord_fact in auxcoord_facts: --> 575 action_build_auxiliary_coordinate(engine, auxcoord_fact) 577 # Detect + process and special 'ukmo' attributes 578 # Run on every cube : they choose themselves whether to trigger. 579 action_ukmo_stash(engine)
File ~/miniforge3/envs/core/lib/python3.12/site-packages/iris/fileformats/_nc_load_rules/actions.py:92, in action_function.
File ~/miniforge3/envs/core/lib/python3.12/site-packages/iris/fileformats/nc_load_rules/actions.py:410, in action_build_auxiliary_coordinate(engine, auxcoord_fact) 407 rule_name += f"{coord_type}" 409 cf_var = engine.cf_var.cf_group.auxiliary_coordinates[var_name] --> 410 hh.build_auxiliary_coordinate(engine, cf_var, coord_name=coord_name) 412 return rule_name
File ~/miniforge3/envs/core/lib/python3.12/site-packages/iris/fileformats/_nc_load_rules/helpers.py:1238, in build_auxiliary_coordinate(engine, cf_coord_var, coord_name, coord_system) 1236 points_data = cf_coord_var.cf_label_data(cf_var) 1237 else: -> 1238 points_data = _get_cf_var_data(cf_coord_var, engine.filename) 1240 # Get any coordinate bounds. 1241 cf_bounds_var, climatological = get_cf_bounds_var(cf_coord_var)
File ~/miniforge3/envs/core/lib/python3.12/site-packages/iris/fileformats/netcdf/loader.py:219, in _get_cf_var_data(cf_var, filename) 217 result = cf_var._data_array 218 else: --> 219 total_bytes = cf_var.size * cf_var.dtype.itemsize 220 if total_bytes < _LAZYVAR_MIN_BYTES: 221 # Don't make a lazy array, as it will cost more memory AND more time to access. 222 # Instead fetch the data immediately, as a real array, and return that. 223 result = cf_var[:]
AttributeError: type object 'str' has no attribute 'itemsize'
Hello @leosaffin Thanks for the bug report.
So, the issue is with the expver variable, which is stored using the "variable length" string type.
If I remove the expver variable from the netCDF file you proposed downloading in the description (using ncks -x -v expver), then Iris can successfully load the data into a cube.
The string types in the attributes are not actually a problem, Iris can decode these fine (although I have admittedly not seen this in an attribute before).
However, I think your proposed solution still stands - I will test.
I think that whilst the proposed solution:
total_bytes = cf_var.size * np.dtype(cf_var.dtype).itemsize
does allow the code to proceed, it is not working entirely as expected as itemsize for a Unicode dtype (which is what dtype(str) resolves to) is returned as 0, so total_bytes will always be calculated as zero for a variable length string.
The issue is that the length of a variable length (VLEN) array cannot be determined until the data has been read from disk, which sort of negates the point of the check that is being made in the netcdf loader here:
https://github.com/SciTools/iris/blob/a01a8c405bf5cb572c637b3fb5c0d6ee15f9b665/lib/iris/fileformats/netcdf/loader.py#L218-L222
This would presumably be the case for any variable length type, not just strings.
Perhaps in the case of VLEN datatypes, we should err on the side of caution and always load lazily?
Perhaps in the case of VLEN datatypes, we should err on the side of caution and always load lazily?
This should be fine. The code you reference was added in response to #5053 - I expect the Venn overlap between workflows affected by #5053 and workflows handling variable length arrays to be small, if it exists at all. And if we do make a couple of workflows slower it may be a necessary sacrifice! Just needs a sensible What's New entry.
We've had reports that the ncks workaround doesn't work 100% of the time. The team are deployed on other priorities at the moment (Backlog · 🐔Iris v3.11 (github.com)), but here is a slightly more sophisticated workaround in the meantime:
from pathlib import Path
from iris.cube import CubeList
from ncdata.iris import to_iris
from ncdata.netcdf4 import from_nc4
from ncdata.threadlock_sharing import enable_lockshare
enable_lockshare(iris=True)
def load_expver_as_int(file_path: Path | str) -> CubeList:
"""Load a NetCDF file and convert the 'expver' variable to an integer.
Iris does not yet support this sort of string variable
([https://github.com/SciTools/iris/issues/6149](https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FSciTools%2Firis%2Fissues%2F6149&data=05%7C02%7Cml-avd-support%40metoffice.gov.uk%7C8b79183c31844ea437bb08dce8842f1c%7C17f1816120d7474687fd50fe3e3b6619%7C0%7C0%7C638640901632719501%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=NOSnN0eVLfkT6T2lPZD1DqF0DWrnlyVJj6mkkm8ZZL4%3D&reserved=0)), but we know that
`expver` is a string representing an integer, so convert it to `int` before
passing to Iris.
"""
file_path = Path(file_path)
dataset = from_nc4(file_path)
if "expver" not in dataset.variables:
raise KeyError(f"Variable 'expver' not found in {file_path}")
new_dtype = int
expver = dataset.variables["expver"]
expver.data = expver.data.astype(new_dtype)
expver.dtype = new_dtype
return to_iris(dataset)
When I try to use this workaround I get this error:
cube = load_expver_as_int(f'{path}ERA5-Land_hourly_pr_tas_td_{year}_{month}.nc')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
line 24, in load_expver_as_int
dataset = from_nc4(file_path)
^^^^^^^^^^^^^^^^^^^
File ".../default-current/lib/python3.11/site-packages/ncdata/netcdf4.py", line 308, in from_nc4
ncdata = _from_nc4_group(nc4ds)
^^^^^^^^^^^^^^^^^^^^^^
File ".../default-current/lib/python3.11/site-packages/ncdata/netcdf4.py", line 264, in _from_nc4_group
var.data = da.from_array(
^^^^^^^^^^^^^^
File ".../default-current/lib/python3.11/site-packages/dask/array/core.py", line 3491, in from_array
chunks = normalize_chunks(
^^^^^^^^^^^^^^^^^
File ".../default-current/lib/python3.11/site-packages/dask/array/core.py", line 3098, in normalize_chunks
chunks = auto_chunks(chunks, shape, limit, dtype, previous_chunks)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../default-current/lib/python3.11/site-packages/dask/array/core.py", line 3272, in auto_chunks
raise ValueError(
ValueError: auto-chunking with dtype.itemsize == 0 is not supported, please pass in chunks explicitly
Hi @mo-cbradsha, I'm afraid I can't replicate your error. This still works for the original file @momichaelkendon sent me, and just to be sure I registered for a CDS account and followed @leosaffin's instructions - works for that file too.
What is different between the downloaded files (whether error or not)
When I try to use this workaround I get this error: ... raise ValueError( ValueError: auto-chunking with dtype.itemsize == 0 is not supported, please pass in
chunksexplicitly
@trexfeathers I'm afraid I can't replicate your error. ... followed @leosaffin's instructions - works for that file too.
OK I've checked this out a bit...
I find that the key difference is that if, as @leosaffin said, "I just ticked the first box for each value", you get a dimensionless variable which looks like this in the dump:
netcdf CDS_newstyle_example {
dimensions:
valid_time = 1 ;
pressure_level = 2 ;
latitude = 721 ;
longitude = 1440 ;
. . .
string expver ;
But if you select multiple timepoints, you get something more like this :
netcdf ERA5-Land_hourly_pr_tas_td_2016_July {
dimensions:
valid_time = 744 ;
latitude = 61 ;
longitude = 91 ;
variables:
. . .
string expver(valid_time) ;
The 'expver' variable is unusual in the new ERA netcdf format, because it has a netcdf4 'string' datatype, which is a variable-length string.
Both Iris and Ncdata are attempting to wrap this in a dask array using dask.array.from_array(variable, chunks="auto") or equivalent.
This fails because in this case variable.dtype is not a numpy dtype, but just Python str. So there is no var.dtype.itemsize.
So , the attempt to use "auto" chunks fails, in different ways for Iris and Ncdata, but with essentially the same cause.
Other ways
Netcdf4
Using Python netCDF4 and Dask in the usual way shows the same problem :
>>> ds = nc4.Dataset(path)
>>> var = ds.variables['expver']
>>> var.dtype
<class 'str'>
>>> import dask.array
>>> darr = dask.array.from_array(var, chunks="auto")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_09_02/lib/python3.12/site-packages/dask/array/core.py", line 3523, in from_array
chunks = normalize_chunks(
^^^^^^^^^^^^^^^^^
File "/net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_09_02/lib/python3.12/site-packages/dask/array/core.py", line 3130, in normalize_chunks
chunks = auto_chunks(chunks, shape, limit, dtype, previous_chunks)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_09_02/lib/python3.12/site-packages/dask/array/core.py", line 3304, in auto_chunks
raise ValueError(
ValueError: auto-chunking with dtype.itemsize == 0 is not supported, please pass in `chunks` explicitly
>>>
Xarray
It turns out that xarray can handle this variable, but I guess it must be pre-scanning the data content to do so, since it treats it as an array of four-character fixed-length strings :
>>> xds = xr.open_dataset(path, chunks="auto")
>>> xds
<xarray.Dataset> Size: 50MB
Dimensions: (valid_time: 744, latitude: 61, longitude: 91)
Coordinates:
number int64 8B ...
* valid_time (valid_time) datetime64[ns] 6kB 2016-07-01 ... 2016-07-31T23:...
* latitude (latitude) float64 488B 26.0 25.9 25.8 25.7 ... 20.2 20.1 20.0
* longitude (longitude) float64 728B 109.0 109.1 109.2 ... 117.8 117.9 118.0
expver (valid_time) <U4 12kB dask.array<chunksize=(744,), meta=np.ndarray>
Data variables:
d2m (valid_time, latitude, longitude) float32 17MB dask.array<chunksize=(744, 61, 91), meta=np.ndarray>
. . .
>>> xds['expver']
<xarray.DataArray 'expver' (valid_time: 744)> Size: 12kB
dask.array<open_dataset-expver, shape=(744,), dtype=<U4, chunksize=(744,), chunktype=numpy.ndarray>
Coordinates:
. . .
Note the "<U4" datatype.
Interim Solutions ?
I think that both ncdata and Iris should be updated to handle these cases, but obviously that is some way off.
In the meantime, perhaps xarray provides a route to fixing it ?
Unfortunately though, ncdata.iris_xarray.cubes_from_xarray() fails in the same way.
However, I find that you can use xarray.DataArray.to_iris() .
But it's a bit fiddly because you must convert individual data variables, and in this particular case we also seem to have a problem due to a "standard_name = "unknown" attribute.
Working around that though, I can get some sense :
>>> xrds = xarray.open_dataset(path, chunks="auto")
>>> xrds
<xarray.Dataset> Size: 50MB
Dimensions: (valid_time: 744, latitude: 61, longitude: 91)
Coordinates:
number int64 8B ...
* valid_time (valid_time) datetime64[ns] 6kB 2016-07-01 ... 2016-07-31T23:...
* latitude (latitude) float64 488B 26.0 25.9 25.8 25.7 ... 20.2 20.1 20.0
* longitude (longitude) float64 728B 109.0 109.1 109.2 ... 117.8 117.9 118.0
expver (valid_time) <U4 12kB dask.array<chunksize=(744,), meta=np.ndarray>
Data variables:
d2m (valid_time, latitude, longitude) float32 17MB dask.array<chunksize=(744, 61, 91), meta=np.ndarray>
t2m (valid_time, latitude, longitude) float32 17MB dask.array<chunksize=(744, 61, 91), meta=np.ndarray>
. . .
>>> var1 = xrds["d2m"]
>>> var1
<xarray.DataArray 'd2m' (valid_time: 744, latitude: 61, longitude: 91)> Size: 17MB
dask.array<open_dataset-d2m, shape=(744, 61, 91), dtype=float32, chunksize=(744, 61, 91), chunktype=numpy.ndarray>
Coordinates:
number int64 8B ...
* valid_time (valid_time) datetime64[ns] 6kB 2016-07-01 ... 2016-07-31T23:...
* latitude (latitude) float64 488B 26.0 25.9 25.8 25.7 ... 20.2 20.1 20.0
* longitude (longitude) float64 728B 109.0 109.1 109.2 ... 117.8 117.9 118.0
expver (valid_time) <U4 12kB dask.array<chunksize=(744,), meta=np.ndarray>
Attributes: (12/32)
. . .
long_name: 2 metre dewpoint temperature
units: K
standard_name: unknown
GRIB_surface: 0.0
>>>
>>> cube = var1.to_iris()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_09_02/lib/python3.12/site-packages/xarray/core/dataarray.py", line 4506, in to_iris
return to_iris(self)
^^^^^^^^^^^^^
File "/net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_09_02/lib/python3.12/site-packages/xarray/convert.py", line 116, in to_iris
cube = iris.cube.Cube(masked_data, **args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_09_02/lib/python3.12/site-packages/iris/cube.py", line 1261, in __init__
self.standard_name = standard_name
^^^^^^^^^^^^^^^^^^
File "/net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_09_02/lib/python3.12/site-packages/iris/common/mixin.py", line 170, in standard_name
self._metadata_manager.standard_name = _get_valid_standard_name(name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/net/project/ukmo/scitools/opt_scitools/conda/deployments/default-2024_09_02/lib/python3.12/site-packages/iris/common/mixin.py", line 44, in _get_valid_standard_name
raise ValueError("{!r} is not a valid standard_name".format(name))
ValueError: 'unknown' is not a valid standard_name
>>>
>>> del var1.attrs["standard_name"]
>>> cube = var1.to_iris()
>>> cube
<iris 'Cube' of 2 metre dewpoint temperature / (K) (time: 744; latitude: 61; longitude: 91)>
>>> print(cube[:3,2,2].data)
[295.819580078125 295.9404296875 296.581787109375]
>>>
So, rather fiddly, but possible.
Does this help at all ??
Extra Note:
It just occurred to me to wonder whether Iris chunk-control might offer a way around this.
E.G. wrap the above problem load with CHUNK_CONTROL.set(validity_time=-1):
Unfortunately that's a no, because this line always runs first, and always fails.
But this aspect should be considered for a future fix. For instance, if we decide to simply set "chunks=-1" for "string" type variables, then we should ensure that manual chunk control is still possible for rare cases which have huge string variables.
Monkey-patch "solution"
We tried a nasty quick fix like this :
import numpy as np
import iris
from iris.fileformats.cf import CFVariable
orig_cfgetattr = CFVariable.__getattr__
def modded_cfgetattr(self, name):
result = orig_cfgetattr(self, name)
if name == "dtype" and result == str:
result = np.dtype('U1')
setattr(self, name, result) # update the cached property
return result
CFVariable.__getattr__ = modded_cfgetattr
It "works" at least to make loading possible. The resulting coord has an numpy object-array as its data, which might give trouble elsewhere (like saving it?), but presumably you could fix that yourself by deleting it or converting to something more standard.
From @SciTools/peloton, we need to decide on the approach for this before we take any actions.
We will probably need an internal meeting, as this affects Met Office users more immediately.
From @SciTools/peloton, we need to decide on the approach for this before we take any actions.
After some more investigation, summary of my take :
- See https://github.com/pp-mo/ncdata/pull/108 for account of how I think dask can't be expected to handle this case "automatically", so Iris does need to deal with it.
- as mentioned above, one obvious hole in our current scheme is that you ought to be able to work around this by taking charge using
iris.file_formats.netcdf.loading.CHUNK_CONTROL. Right now that fails, which we can probably say is a bug which we should certainly fix This might be all that we do - the REALLY big question, for me, is whether we can implement any sensible automatic resolution of this. At least, without pre-scanning the offending variable-length-so-just-dont-know-how-big-it-might-be thing.
Possible choices for automatic (no user opt-in) resolution
The whole point of Iris attempting to choose chunks automatically (and Dask too for that matter), is that it attempts to choose chunks big enough to be efficient while (hopefully) not so big as to blow memory. Unknown sizes do kind-of wreck that.
With an unlimited dimension the loader is still in control of how much data it fetches, but a variable-length datatype is just a big unknown.
Regarding automatic solutions, I can see only a few possible choices to "guess" an item-size. For a numeric, we can perhaps guess a small-ish number like maybe 1 or 10-20, which could also work for "typical" short strings (like station numbers?). Though, strings also have a possible complicating factor if they actually use unicode which may have an extra amplifying factor. (It totally depends on why you think someone might choose to use a variable-length storage type. So maybe that's just a typical set of numbers within a range, not on an additional dimension. So one could expect that to be 10's rather than 100's ??)
If we implement this, I guess we should probably issue a warning when it happens, and perhaps we can also detect + warn when an actual chunk is bigger than was assumed (but I'm not sure we can manage this, because of different execution context when fetch occurs).
Possibly we can also supply controls to adjust the automatic "itemsize default" -- but of course you can always take charge of the whole chunking decisions via CHUNK_CONTROL, in which case none of this matters (and that is frankly preferable).
I don't think there is anything in the netCDF4 interface that allows you to get the max length without actually loading the whole variable. AFAICT even if you only fetch one array element (i.e. one string, or set of numbers), you never know how much space you may be committing to. Nevertheless, though costly, we could instead just fetch all the data to determine the max-length. It does look like xarray may be doing that.
ping @ukmo-ccbunney Lots of the above from yesterday's discussion with you. All errors + exceptions my own !! What do you think ??
Since this is a pretty big flaw in iris for users (CDS data is only becoming more used) I think a hacky solution now (or this release) is better than a perfectly considered solution later (yes, I know there's nothing more permament than a temperory solution, but not working on a huge class of external data is a big deal).
Given we can make sure we're only guessing the chunk size for string coordinates, I don't think it's too much of a stretch to get all the data to determine chunk size. String coordinates aren't large, and slowing down cases of loading lots of small files with many string coords is worth having CDS files just-work. It's not something I think there's a large demand for a warning for, given users already complain about warning fatigue with iris.
Given we can make sure we're only guessing the chunk size for string coordinates, I don't think it's too much of a stretch to get all the data to determine chunk size. String coordinates aren't large,
I think this misses a possibly somewhat big point : the "expver" variable is really not a coordinate -- except in xarray's slightly odd way of looking at things : It is actually a data variable in its own right, which would make a cube, if we could only load it.
Since this is a pretty big flaw in iris for users (CDS data is only becoming more used) I think a hacky solution now (or this release) is better than a perfectly considered solution later (yes, I know there's nothing more permament than a temperory solution, but not working on a huge class of external data is a big deal).
Although I kind-of disagreed about the "it's only a coordinate" part, I'm increasingly agreeing with this that we should have an automatic solution. Big hint : This is already labelled for "3.12" !
Yea I can see from above you've already put alot of thought into this problem. I just thought it would be putting in my hat as a user that there's a big need for this to be automated solution of some kind for the next release, so you get some feedback on the issue.
given users already complain about warning fatigue with iris.
Filtering Warnings — Iris 3.11.1 documentation
NOTE: we must be able to fix this ALSO by controlling chunking. That also doesn't work at present..
The actual problem is that this line is always run (and always fails), even when a user chunking override means the value is not needed.
Presumably with a little more intelligence we can deduce that we don't need that, or at least we can somehow handle the fact that ".dtype is str and doesn't have a .itemsize", in the case that a relevant chunking override is provided.
Agreed solution between @pp-mo, @ukmo-ccbunney, @trexfeathers: align the handling of variable length strings with all our variable lengths, which has proven fine for almost all use cases in the past.