eofs
eofs copied to clipboard
SVD error when using eof solver on dask array
I have been getting an SVD error when trying to call the xarray eof solver on a daskarray.
The error is due to the following line in eofs.standard:
nonMissingIndex = np.where(np.logical_not(np.isnan(self._data[0])))[0]
np.where always fails and gives nans for dask arrays (see e.g. https://stackoverflow.com/questions/59957541/what-is-the-dask-equivalent-of-numpy-where)
Possibly related to #115 (although I can no longer see those notebooks).
Solved by calling .load() before calling the solver, but loses the advantages of dask.
I had the same problem and I was able to solve it by replacing this line in standard.py
nonMissingIndex = np.where(np.logical_not(np.isnan(self._data[0])))[0]
by
nonMissingIndex = dask.array.where(np.logical_not(np.isnan(self._data[0])))[0]
nonMissingIndex.compute_chunk_sizes()
I think it doesn't lose the advantages of dask as I have a dataset that doesn't fit in memory but the code worked even so. I hope that can be useful to anyone.