Investigate causes of regridding performance issue

[lbdreyer]

Some users have cited slow downs in regridding as a blocker to moving to Iris 2 (and therefore Python 3 as they don't have access to Iris 1.13 in a Python 3 environment).

This work is focussed on a single, high priority usecase. We need to investigate this usecase to identify the cause of the slowdown. We can address the other usecases at a later stage, but they are not a part of our MVP for Iris 3.

The outcome of this work should be a proposal of what needs addressing in the next sprint.

NOTE: this will be a timeboxed activity: work on this issue will be limited to two person days

---

Acceptance Criteria

• [ ] The time that we agreed to spend on this has been reached
• [ ] A new issue proposing what changes (if any) are needed

[bjlittle]

Note that, this may be related to the slow down caused by processing masked data, see #3470. Those using iris version 1.13 will be pinned to an older version of numpy.

For this investigation, we should be open to the possibility that the bottleneck may be (partly?) due to changes in one or more of the package dependencies of iris, as well as iris itself.

[abooton]

I investigated the difference in regridding data using linear regrid, for data in iris 1.13 vs iris 2.1 (following various reports of slow downs):

• Regridding 2D or 3D iris cube data is takes the same time.
• Regridding 3D data takes ~10secs, but regridding masked 3D array takes ~20secs. Times are consistent between versions.

(The regridding routines have not changed so this is as expected)

However, when the regridding routine is used:

• The data is realised. This can be significantly slower than before. (In the StaGE data example, this slow down was from 9sec to 70secs. It is a known issue due to dask disliking tiny chunks)
• Additionally, the netCDF files now load data as masked arrays by default.

Following up on the "slow down" comments also established that:

• Stratify is too slow for use in StaGE because it uses float64 instead of float32. They therefore bypassed stratify to use float32. (They were going to put up a PR to make the changes but it is non-trivial due to the inhertence structure.)
• Other StaGE regridding slow down issues have (almost) been isolated to a bug fix in the multiprocessing module exposing an issue in StaGE's lad-sea regrid routine. (In which coastlines are regridding by looping through the 24000 coastline points, extracting single cubes of the coastline region and are regridded using area-weighted regrid). This is being investigated by the StaGE team.

[bjlittle]

@abooton Thanks. Nice work.

Could you please raise an appropriate issue on python-stratify, so that we don't forget about this, thanks 👍

[abooton]

The stratify issue is: https://github.com/SciTools-incubator/python-stratify/issues/38

[wjbenfold]

Are there examples of this issue (/these issues) that can be tried against the performance improvements in PR #4400 to check what else needs doing?

[wjbenfold]

Progressing this issue is currently blocked by a lack of information about whether it's already been sufficiently addressed by #4400

[trexfeathers]

Estimated remaining work is due diligence for whether this can be closed

[github-actions[bot]]

In order to maintain a backlog of relevant issues, we automatically label them as stale after 500 days of inactivity.

If this issue is still important to you, then please comment on this issue and the stale label will be removed.

Otherwise this issue will be automatically closed in 28 days time.

[github-actions[bot]]

This stale issue has been automatically closed due to a lack of community activity.

If you still care about this issue, then please either:

• Re-open this issue, if you have sufficient permissions, or
• Add a comment stating that this is still relevant and someone will re-open it on your behalf.