ecmwf
Large grib error: Invalid argument
What happened?
I wanted to open a 4.5GB grib file. It does work to load it but accessing certain variables beyond a particular scope does not work.
I checked the file by using pygrib. The data is there and no issues are to be found if i use a message index that corresponds to the first non-working value.
What are the steps to reproduce the bug?
ds = xr.open_dataset("data/era5/1940_2.grib")
ds['u10'].isel(time=4387) *1
I have investigated this a bit and find that the value put in to the method is -5 (See error log output below). This seems to be the value for all contents of self.field_id_index if you go through the items after value at index 4387.
Coincidentally the value before it turns -5 is 2147017176 with next values preceding this value just having added 489516 per stride. This would lead to 2147506692 as the consequentially next value. Which is larger than 2**31 while the value before was slightly below.
I have a strong suspicion that somewhere a int32 is being used. Since im no cfgrib expert i hope you can take a look - i am sure others would also benefit!
(I am on windows, python 3.10)
Version
cfgrib.version == '0.9.15.0'
Platform (OS and architecture)
Windows 11 24H2 64bit
Relevant log output
---------------------------------------------------------------------------
OSError Traceback (most recent call last)
Cell In[16], line 2
1 for i in range(8000):
----> 2 ds['u10'].isel(time=i) ** 2
File d:\miniforge3\lib\site-packages\xarray\core\_typed_ops.py:564, in DataArrayOpsMixin.__pow__(self, other)
563 def __pow__(self, other: DaCompatible) -> Self | Dataset | DataTree:
--> 564 return self._binary_op(other, operator.pow)
File d:\miniforge3\lib\site-packages\xarray\core\dataarray.py:4824, in DataArray._binary_op(self, other, f, reflexive)
4820 other_variable_or_arraylike: DaCompatible = getattr(other, "variable", other)
4821 other_coords = getattr(other, "coords", None)
4823 variable = (
-> 4824 f(self.variable, other_variable_or_arraylike)
4825 if not reflexive
4826 else f(other_variable_or_arraylike, self.variable)
4827 )
4828 coords, indexes = self.coords._merge_raw(other_coords, reflexive)
4829 name = result_name([self, other])
File d:\miniforge3\lib\site-packages\xarray\core\_typed_ops.py:950, in VariableOpsMixin.__pow__(self, other)
949 def __pow__(self, other: VarCompatible) -> Self | T_DA | Dataset | DataTree:
--> 950 return self._binary_op(other, operator.pow)
File d:\miniforge3\lib\site-packages\xarray\core\variable.py:2326, in Variable._binary_op(self, other, f, reflexive)
2324 other_data, self_data, dims = _broadcast_compat_data(other, self)
2325 else:
-> 2326 self_data, other_data, dims = _broadcast_compat_data(self, other)
2327 keep_attrs = _get_keep_attrs(default=False)
2328 attrs = self._attrs if keep_attrs else None
File d:\miniforge3\lib\site-packages\xarray\core\variable.py:2942, in _broadcast_compat_data(self, other)
2939 dims = new_self.dims
2940 else:
2941 # rely on numpy broadcasting rules
-> 2942 self_data = self.data
2943 other_data = other
2944 dims = self.dims
File d:\miniforge3\lib\site-packages\xarray\core\variable.py:416, in Variable.data(self)
414 return self._data
415 elif isinstance(self._data, indexing.ExplicitlyIndexed):
--> 416 return self._data.get_duck_array()
417 else:
418 return self.values
File d:\miniforge3\lib\site-packages\xarray\core\indexing.py:836, in MemoryCachedArray.get_duck_array(self)
835 def get_duck_array(self):
--> 836 self._ensure_cached()
837 return self.array.get_duck_array()
File d:\miniforge3\lib\site-packages\xarray\core\indexing.py:833, in MemoryCachedArray._ensure_cached(self)
832 def _ensure_cached(self):
--> 833 self.array = as_indexable(self.array.get_duck_array())
File d:\miniforge3\lib\site-packages\xarray\core\indexing.py:790, in CopyOnWriteArray.get_duck_array(self)
789 def get_duck_array(self):
--> 790 return self.array.get_duck_array()
File d:\miniforge3\lib\site-packages\xarray\core\indexing.py:653, in LazilyIndexedArray.get_duck_array(self)
649 array = apply_indexer(self.array, self.key)
650 else:
651 # If the array is not an ExplicitlyIndexedNDArrayMixin,
652 # it may wrap a BackendArray so use its __getitem__
--> 653 array = self.array[self.key]
655 # self.array[self.key] is now a numpy array when
656 # self.array is a BackendArray subclass
657 # and self.key is BasicIndexer((slice(None, None, None),))
658 # so we need the explicit check for ExplicitlyIndexed
659 if isinstance(array, ExplicitlyIndexed):
File d:\miniforge3\lib\site-packages\cfgrib\xarray_plugin.py:163, in CfGribArrayWrapper.__getitem__(self, key)
159 def __getitem__(
160 self,
161 key: xr.core.indexing.ExplicitIndexer,
162 ) -> np.ndarray:
--> 163 return xr.core.indexing.explicit_indexing_adapter(
164 key, self.shape, xr.core.indexing.IndexingSupport.BASIC, self._getitem
165 )
File d:\miniforge3\lib\site-packages\xarray\core\indexing.py:1014, in explicit_indexing_adapter(key, shape, indexing_support, raw_indexing_method)
992 """Support explicit indexing by delegating to a raw indexing method.
993
994 Outer and/or vectorized indexers are supported by indexing a second time
(...)
1011 Indexing result, in the form of a duck numpy-array.
1012 """
1013 raw_key, numpy_indices = decompose_indexer(key, shape, indexing_support)
-> 1014 result = raw_indexing_method(raw_key.tuple)
1015 if numpy_indices.tuple:
1016 # index the loaded duck array
1017 indexable = as_indexable(result)
File d:\miniforge3\lib\site-packages\cfgrib\xarray_plugin.py:172, in CfGribArrayWrapper._getitem(self, key)
167 def _getitem(
168 self,
169 key: T.Tuple[T.Any, ...],
170 ) -> np.ndarray:
171 with self.datastore.lock:
--> 172 return self.array[key]
File d:\miniforge3\lib\site-packages\cfgrib\dataset.py:373, in OnDiskArray.__getitem__(self, item)
371 continue
372 # NOTE: fill a single field as found in the message
--> 373 message = self.index.get_field(message_ids[0]) # type: ignore
374 values = get_values_in_order(message, array_field[tuple(array_field_indexes)].shape)
375 array_field.__getitem__(tuple(array_field_indexes)).flat[:] = values
File d:\miniforge3\lib\site-packages\cfgrib\messages.py:488, in FieldsetIndex.get_field(self, message_id)
487 def get_field(self, message_id: T.Any) -> abc.Field:
--> 488 return ComputedKeysAdapter(self.fieldset[message_id], self.computed_keys)
File d:\miniforge3\lib\site-packages\cfgrib\messages.py:345, in FileStream.__getitem__(self, item)
343 def __getitem__(self, item: T.Optional[OffsetType]) -> Message:
344 with open(self.path, "rb") as file:
--> 345 return self.message_from_file(file, offset=item)
File d:\miniforge3\lib\site-packages\cfgrib\messages.py:341, in FileStream.message_from_file(self, file, offset, **kwargs)
339 def message_from_file(self, file, offset=None, **kwargs):
340 # type: (T.IO[bytes], T.Optional[OffsetType], T.Any) -> Message
--> 341 return Message.from_file(file, offset, **kwargs)
File d:\miniforge3\lib\site-packages\cfgrib\messages.py:94, in Message.from_file(cls, file, offset, **kwargs)
92 offset, field_in_message = offset
93 if offset is not None:
---> 94 file.seek(offset)
95 codes_id = None
96 if field_in_message == 0:
OSError: [Errno 22] Invalid argument
Accompanying data
Dont have a link sorry.
Organisation
Stockholms University
@iainrussell I see that no bug or issues have received in a month but that you are still active on this repository. Just out of curiosity, there is no reason to suspect that cfgrib is not being actively (issuebased) maintained ?
Mhh i could not wait. after a lot of digging i found the source.
it is in gribapi.py around lines 956 for me. It makes a long* on windows this is a 32bit integer. zonk.
`@require(msgid=int, key=str) def grib_get_long(msgid, key): """ @brief Get the value of a key in a message as an integer.
@param msgid id of the message loaded in memory
@param key key name
@return value of key as int
@exception CodesInternalError
"""
h = get_handle(msgid)
value_p = ffi.new("long*")
err = lib.grib_get_long(h, key.encode(ENC), value_p)
GRIB_CHECK(err)
return value_p[0]`
I am now writing a long long* in there to see if it helps.
Nuh... That is still true but it somehow lies deeper as well.
I rewrote the code to virtually look up a double and then cast it into an int. but the -5 still gets returned.
So this seems to be a deeper issue where this -5 comes from.
In my debugger is see that the int -5 is named GRIB_7777_NOT_FOUND.
It seeems that the first message this happens at is 2011692580928.
Does that mean that get_handle already had an issue?
Well in other news - i traced out where things go haywire a bit more. its funny that pygrib can read the file but cfgrib is not able to even though both use eccodes.
How do I get in contact with the eccodes developers? It seems to be on a confluence page https://confluence.ecmwf.int/display/ECC/grib_copy
But unfortunately i have no access there.
Windows is not a platform that is used for technical work at ECMWF. Therefore we will not be able to support this platform as well as we do for others, such as Linux
Please see https://confluence.ecmwf.int/pages/viewpage.action?pageId=131394516
@dtip Maybe you can weigh in here - i also mailed oxidian. Sorry about double posting, I just learned that you are available on gh too.