xarray-spatial icon indicating copy to clipboard operation
xarray-spatial copied to clipboard

getting error when trying to run look <source elided> out = np.full(arr1.shape, np.nan, dtype=np.float32) rows, cols = arr1.shape

Open krishnaglodha opened this issue 1 year ago • 3 comments

Describe the bug I'm trying to get statistical values of my xarray.DataArray in loop , but getting the error

TypingError                               Traceback (most recent call last)
Cell In[18], [line 2](vscode-notebook-cell:?execution_count=18&line=2)
      [1](vscode-notebook-cell:?execution_count=18&line=1) for val in index_xarray:
----> [2](vscode-notebook-cell:?execution_count=18&line=2)     print(val.max().compute().values.item())

File ~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1169, in DataArray.compute(self, **kwargs)
   [1150](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1150) """Manually trigger loading of this array's data from disk or a
   [1151](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1151) remote source into memory and return a new array. The original is
   [1152](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1152) left unaltered.
   (...)
   [1166](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1166) dask.compute
   [1167](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1167) """
   [1168](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1168) new = self.copy(deep=False)
-> [1169](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1169) return new.load(**kwargs)

File ~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1143, in DataArray.load(self, **kwargs)
   [1125](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1125) def load(self, **kwargs) -> Self:
   [1126](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1126)     """Manually trigger loading of this array's data from disk or a
   [1127](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1127)     remote source into memory and return this array.
   [1128](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1128) 
   (...)
   [1141](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1141)     dask.compute
   [1142](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1142)     """
-> [1143](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1143)     ds = self._to_temp_dataset().load(**kwargs)
   [1144](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1144)     new = self._from_temp_dataset(ds)
   [1145](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1145)     self._variable = new._variable

File ~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataset.py:845, in Dataset.load(self, **kwargs)
    [842](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataset.py:842) chunkmanager = get_chunked_array_type(*lazy_data.values())
    [844](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataset.py:844) # evaluate all the chunked arrays simultaneously
--> [845](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataset.py:845) evaluated_data: tuple[np.ndarray[Any, Any], ...] = chunkmanager.compute(
    [846](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataset.py:846)     *lazy_data.values(), **kwargs
    [847](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataset.py:847) )
    [849](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataset.py:849) for k, data in zip(lazy_data, evaluated_data):
    [850](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataset.py:850)     self.variables[k].data = data

File ~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/namedarray/daskmanager.py:86, in DaskManager.compute(self, *data, **kwargs)
     [81](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/namedarray/daskmanager.py:81) def compute(
     [82](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/namedarray/daskmanager.py:82)     self, *data: Any, **kwargs: Any
     [83](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/namedarray/daskmanager.py:83) ) -> tuple[np.ndarray[Any, _DType_co], ...]:
     [84](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/namedarray/daskmanager.py:84)     from dask.array import compute
---> [86](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/namedarray/daskmanager.py:86)     return compute(*data, **kwargs)

File ~/anaconda3/envs/stac/lib/python3.12/site-packages/dask/base.py:661, in compute(traverse, optimize_graph, scheduler, get, *args, **kwargs)
    [658](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/dask/base.py:658)     postcomputes.append(x.__dask_postcompute__())
    [660](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/dask/base.py:660) with shorten_traceback():
--> [661](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/dask/base.py:661)     results = schedule(dsk, keys, **kwargs)
    [663](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/dask/base.py:663) return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])

File ~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:468, in _DispatcherBase._compile_for_args(self, *args, **kws)
    [464](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:464)         msg = (f"{str(e).rstrip()} \n\nThis error may have been caused "
    [465](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:465)                f"by the following argument(s):\n{args_str}\n")
    [466](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:466)         e.patch_message(msg)
--> [468](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:468)     error_rewrite(e, 'typing')
    [469](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:469) except errors.UnsupportedError as e:
    [470](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:470)     # Something unsupported is present in the user code, add help info
    [471](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:471)     error_rewrite(e, 'unsupported_error')

File ~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:409, in _DispatcherBase._compile_for_args.<locals>.error_rewrite(e, issue_type)
    [407](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:407)     raise e
    [408](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:408) else:
--> [409](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:409)     raise e.with_traceback(None)

TypingError: Failed in nopython mode pipeline (step: nopython frontend)
('wrong tuple length for $130load_attr.11: ', 'expected 2, got 3')
During: typing of exhaust iter at /Users/krishnaglodha/anaconda3/envs/stac/lib/python3.12/site-packages/xrspatial/multispectral.py (772)

File "../../../../anaconda3/envs/stac/lib/python3.12/site-packages/xrspatial/multispectral.py", line 772:
def _normalized_ratio_cpu(arr1, arr2):
    <source elided>
    out = np.full(arr1.shape, np.nan, dtype=np.float32)
    rows, cols = arr1.shape
    ^

Expected behavior I should be able to print max value Code

catalog = pystac_client.Client.open('https://earth-search.aws.element84.com/v1')

time_range = f"{start_date}/{end_date}"
collection = "sentinel-2-l2a"
geometry = polygon["features"][0]["geometry"]
search = catalog.search(
    collections=[collection], intersects=geometry, datetime=time_range
) 
print(search.get_all_items())
data = load(
            search.items(),
            geopolygon=geometry,
            chunks={},
            resolution=1,
            projection="epsg:3857",
        )
dc = data.rio.clip([geometry], crs="epsg:4326")
index_xarray = ms.ndvi(dc.nir, dc.red)

for val in index_xarray:
    print(val.max().compute().values.item())

Desktop (please complete the following information):

  • OS: MacOS
  • Browser - chrome

Additional context Add any other context about the problem here.

krishnaglodha avatar May 25 '24 06:05 krishnaglodha

@krishnaglodha thanks for submitting the issue. I'll take a deeper look at NDVI and see if anything changed with recent 3.12 xarray-spatial release.

Can you tell post what type dc.nir.data is...and dtype, and shape?

What happens if you make sure dc.nir etc are 2d arrays by squeezing out any empty dimensions?

brendancol avatar May 27 '24 03:05 brendancol

Screenshot 2024-05-27 at 10 30 32 AM

what baffled me is that is worked when I passed

data = load(
            search.items(),
            geopolygon=geometry,
            chunks={},
            resolution=1,
            projection="epsg:3857",
        ).isel(time=0)

i.e. for time =0

but when I try to get

data = load(
            search.items(),
            geopolygon=geometry,
            chunks={},
            resolution=1,
            projection="epsg:3857",
        )

and then loop through data , I get error

krishnaglodha avatar May 27 '24 05:05 krishnaglodha

@krishnaglodha you may be triggering a computation when running isel...I'm not totally sure.

It sounds like there may be one layer in the response that does not have the dimensionality that you expect. If you loop through the layers and print each time step out, do you see that there are only two dimensions?

brendancol avatar May 27 '24 16:05 brendancol