Add support for opening a dataset memory without writing it to disc

[labarababa]

Is it possible to read a Dataset from a Zipfile without writing to disc?

This works like a charm:

with gzip.open('some_file', "rb") as f_in:
     with open("tmp", "wb") as f_out:
           shutil.copyfileobj(f_in, f_out)
grib_files = cfgrib.open_datasets("tmp", backend_kwargs={"indexpath": ""})

I want something like that:

with gzip.GzipFile('some_file', 'rb') as zipfile:
    bytes_content = zipfile.read()
grib_files = cfgrib.open_datasets(zipfile, backend_kwargs={"indexpath": ""})

But i am getting the following error:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 4: invalid start byte

Any ideas?

Kind Regards.

[alexamici]

@labarababa I see the point for reading a GRIB from a string or from an opened file, however it is not a trivial update and I don't see it as high priority. Anyway I'll keep the issue open as a feature request.

[Plantain]

This is something that would be useful to me as well - the loading from memory part rather than the zip part. Currently we download GRIB2 files to memory with python, then write them out to disk solely in order to be able to open them with cfgrib. Perhaps this bug should be renamed as I'm not sure the gzip part is relevant. If you give me some pointers I will take a look at getting this implemented.

[alexamici]

Note that this is really a limitation in either ecCodes or in the internal eccodes bindings. In the near feature I'll switch to use the new ecmwf/eccodes-python package to bind to ecCodes and this feature request will belong there.

[juhi24]

I'm also looking forward for this feature. In the best case scenario, cfgrib.read_datasets would accept a file-like object. In addition to in-memory objects, this could enable stuff like reading files directly from S3 object storage using boto3.

[TomAugspurger]

I think https://github.com/ecmwf/eccodes-python/issues/25 is the relevant issue in eccodes-python. https://confluence.ecmwf.int/display/UDOC/How+do+I+decode+messages+from+a+byte+stream+-+ecCodes+FAQ is maybe relevant.