xarray-sentinel icon indicating copy to clipboard operation
xarray-sentinel copied to clipboard

Chunking works only with dask installed

Open corrado9999 opened this issue 3 years ago • 5 comments

I am not sure if this is a bug and, if so, of what.

>>> path = "tests/data/S1B_IW_SLC__1SDV_20210401T052622_20210401T052650_026269_032297_EFA4.SAFE/manifest.safe"
>>> group = 'IW1/R168-N471-E0118'
>>> xs.sentinel1.open_dataset(path, group=group)
<xarray.Dataset>
Dimensions:           (azimuth_time: 1501, slant_range_time: 21632)
Coordinates:
    line              (azimuth_time) float64 0.5 1.5 2.5 ... 1.5e+03 1.5e+03
    pixel             (slant_range_time) float64 0.5 1.5 ... 2.163e+04 2.163e+04
  * azimuth_time      (azimuth_time) datetime64[ns] 2021-04-01T05:26:24.20999...
  * slant_range_time  (slant_range_time) float64 0.005343 0.005343 ... 0.005679
Data variables:
    VH                (azimuth_time, slant_range_time) complex128 ...
    VV                (azimuth_time, slant_range_time) complex128 ...
Attributes: (12/14)
    constellation:              sentinel-1
    platform:                   sentinel-1b
    instrument:                 ['c-sar']
    sat:orbit_state:            descending
    sat:absolute_orbit:         26269
    sat:relative_orbit:         168
    ...                         ...
    sar:instrument_mode:        IW
    sar:polarizations:          ['VV', 'VH']
    sar:product_type:           SLC
    xs:instrument_mode_swaths:  ['IW1', 'IW2', 'IW3']
    Conventions:                CF-1.7
    history:                    created by xarray_sentinel-999

>>> xs.sentinel1.open_dataset(path, group=group, chunks=1000)
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-5-79175c680e53> in <module>
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-7-90f5963f6ea0> in <module>
----> 1 xs.sentinel1.open_dataset(path, group=group, chunks=1000)

~\xarray-sentinel\xarray_sentinel\sentinel1.py in open_dataset(product_urlpath, drop_variables, group, chunks, fs)
    384         else:
    385             annotation_path = list(groups[group]["annotation_path"].values())[0]
--> 386             ds = open_burst_dataset(
    387                 manifest_path,
    388                 measurement_paths=groups[group]["measurement_path"],

~\xarray-sentinel\xarray_sentinel\sentinel1.py in open_burst_dataset(manifest_path, burst_position, measurement_paths, annotation_path, chunks)
    297     data_vars = {}
    298     for pol, data_path in measurement_paths.items():
--> 299         arr = rioxarray.open_rasterio(data_path, chunks=chunks)
    300
    301         arr = arr.squeeze("band").drop_vars(["band", "spatial_ref"])

~\miniconda3\envs\xr-sentinel\lib\site-packages\rioxarray\_io.py in open_rasterio(filename, parse_coordinates, chunks, cache, lock, masked, mask_and_scale, variable, group, default_name, **open_kwargs)
    863
    864     if chunks is not None:
--> 865         result = _prepare_dask(result, riods, filename, chunks)
    866
    867     # Make the file closeable

~\miniconda3\envs\xr-sentinel\lib\site-packages\rioxarray\_io.py in _prepare_dask(result, riods, filename, chunks)
    573     """
    574     # pylint: disable=import-outside-toplevel
--> 575     from dask.base import tokenize
    576
    577     # augment the token with the file modification time

ModuleNotFoundError: No module named 'dask'

corrado9999 avatar Apr 27 '21 22:04 corrado9999

We could:

  1. Add can add a check to skip tests with dask if dask is not installed.
  2. Improve the message error message. That should be done in rioxarray

aurghs avatar Apr 28 '21 06:04 aurghs

This is indeed the default behaviour of xarray. But I agree that the error message is awful.

@aurghs I propose to do in the reverse order:

  1. open a PR to rioxarray where it raises an exception with a better error message
  2. (if 1. fails) add the same check in xarray-sentinel

alexamici avatar Apr 28 '21 10:04 alexamici

This is indeed the default behaviour of xarray. But I agree that the error message is awful.

1 and 2 are not an aut-aut. I would:

  • Add a check to skip tests (pyest) with dask if dask is not installed.
  • Add dask in setup.cfg as optional
  • open a PR to rioxarray where it raises an exception with a better error message

aurghs avatar Apr 28 '21 10:04 aurghs

  1. we do install dask for testing already, no need to test without dask, we don't have any dask related logic to test. (I propose no action)
  2. dask is not a needed dependency so it must not be listed in the install_requires (no action)
  3. we agree, @aurghs would you please take care of it?

alexamici avatar Apr 28 '21 14:04 alexamici

  1. we could add an xfails on the tests with dask
  2. I agree
  3. I think @corrado9999 was preparing the PR.

aurghs avatar Apr 28 '21 14:04 aurghs