xarray icon indicating copy to clipboard operation
xarray copied to clipboard

Loading a reduced dataset by calling an OpenDAP server-side function?

Open jondoesntgit opened this issue 4 years ago • 3 comments
trafficstars

The OpenDAP spec provides for server-side functions to aggregate and reduce file size prior to transmission over the network see here. Pydap offers an API to write custom server-side functions (See source code here and here).

I work a lot with time-series data from instruments in my laboratory. I typically will sample at 10-1000 Hz, but when I analyze the data, it's easier to see long-term drift by resampling the data at ~1 minute. This reduces the size of the dataset by a factor of 600-60k. I'm thinking about writing some plugins for Pydap using their server-side-function API. My question is whether or not xarray supports calling OpenDAP server-side functions; if not, is there a plan to provide for this?

I'm kind of new to OpenDAP, and haven't gotten my hands too dirty yet. All I've done is install a Pydap server, and load a PyTables file. I'd love to use xarray more in my workflow.

jondoesntgit avatar Feb 26 '21 21:02 jondoesntgit

Is this still relevant?

max-sixty avatar Sep 18 '24 18:09 max-sixty

xarray supports calling OpenDAP server-side functions; if not, is there a plan to provide for this?

This would be nice to add to the backend somehow.

cc @Mikejmnez

dcherian avatar Sep 18 '24 18:09 dcherian

Thanks for including me on this! On the client side, yes pydap can escape characters on URLs that allow for serve-side subsetting. And so one can construct URLs with constraint expressions (CEs) to subset the dataset (say on index space) when opening a dataset via xarray. In that scenario all computation/subsetting is done by the server local to the data.

Mikejmnez avatar Sep 18 '24 21:09 Mikejmnez