Lazy rectilinear interpolator

[fnattino]

🚀 Pull Request

Description

Different take to enable the rectilinear interpolator to run lazily #6002 .

Trying to address the same issue as #6006, but there I have made the underlying _RegularGridInterpolator from _scipy_interpolate to run on lazy data, which required switching from scipy.sparse to sparse (not ideal since it would add numba as a dependency).

Here I have tried instead to implement a similar approach as used for regridding, which works on lazy data as well. The downside is that the chunks in the dimensions we are interpolating over need to be merged, but at least we could run interpolation in parallel over the chunks in the other dimensions (and we do not need to add extra dependencies to iris).

[bouweandela]

Nice to see this progressing @fnattino! Did you notice that CI is failing on this pull request?

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 89.85%. Comparing base (7cf9c5f) to head (99f4520). Report is 71 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #6084   +/-   ##
=======================================
  Coverage   89.85%   89.85%           
=======================================
  Files          88       88           
  Lines       23395    23396    +1     
  Branches     4358     4355    -3     
=======================================
+ Hits        21021    21023    +2     
  Misses       1646     1646           
+ Partials      728      727    -1     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[trexfeathers]

@fnattino just want to reassure you that I have been looking at this, but since I have never worked with our interpolators before it is slow progress. Having another go this afternoon with help from some coffee ☕

[fnattino]

No worries @trexfeathers, but thanks for the heads-up! :)

[fnattino]

Thanks a lot for the effort of getting back to this @trexfeathers ! I should have time to get back to this next week, but I will keep you posted!

[fnattino]

Thanks again for the review @trexfeathers and sorry with the delay in replying. In addition to replying to your comments, I have updated a couple of docstrings (where I left comments). I have also realized that the user guide should probably also be updated, especially the section on caching the interpolator? I can take care of this tomorrow. Any other docs that should be updated?

[fnattino]

I have made some small updates to the userguide to add that the linear and nearest-neighbour interpolators can now work with lazy data as well.

Just wanted to double check, but I think we should also remove the subsections on "caching an interpolator" and "caching a regridder": as discussed in the thread below this comment, almost all the work is done when calling the interpolator (within RectilinearInterpolator._interpolate) so there seem to be no actual benefit in constructing and reusing the interpolator objects? And I think this is also true for the regridder, for which all the work seems to done within RectilinearRegridder._regrid?

PS: the linkchecker action seems to fail for something unrelated - the website https://fesom.de (linked here) seems to be unreachable.