xarray icon indicating copy to clipboard operation
xarray copied to clipboard

Expose `memory` argument for "netcdf4" engine

Open ianliu opened this issue 1 year ago • 7 comments

This commit exposes the memory argument for the "netcdf4" engine, allowing one to create a netcdf dataset from a memory buffer, like so:

buffer: bytes = ...
ds = xr.open_dataset("", engine="netcdf4", memory=buffer)
  • [x] Closes #6955

ianliu avatar Aug 26 '22 14:08 ianliu

Thanks @ianliu you should be able to do this with xr.open_dataset(..., backend_kwargs={"memory": buffer})

dcherian avatar Aug 26 '22 15:08 dcherian

@dcherian backend_kwargs doesn't work either, because the NetCDF4BackendEntrypoint.open_dataset function doesn't accept a memory argument:

https://github.com/pydata/xarray/blob/434f9e8929942afc2380eab52a07e77d30cc7885/xarray/backends/netCDF4_.py#L534-L552

ianliu avatar Aug 26 '22 15:08 ianliu

Ah thanks. I always forget that.

Can you add a test please? Somewhere in this class would be the right place I think https://github.com/pydata/xarray/blob/434f9e8929942afc2380eab52a07e77d30cc7885/xarray/tests/test_backends.py#L1225

dcherian avatar Aug 26 '22 16:08 dcherian

@dcherian I've added a unit test, but outside the class. Is this a problem? I see in the Xarray's contributing page that Xarray is transitioning to more functional tests.

ianliu avatar Aug 26 '22 17:08 ianliu

Hmm, the test doesn't pass like this, it is complaining that nc4 is not defined. I will put the test inside the class.

ianliu avatar Aug 26 '22 17:08 ianliu

@dcherian can you review my changes? I think the PR is done now.

ianliu avatar Aug 26 '22 18:08 ianliu

I've just committed a change that also allows one to save a netcdf file to memory, like so:

ds = xr.Dataset({ "v": xr.DataArray(data=range(10)) })
buf = ds.to_netcdf(engine="netcdf4")
assert xr.open_dataset("", engine="netcdf4", memory=buf).equals(ds)

ianliu avatar Aug 30 '22 19:08 ianliu