aeon [MNT] CI add pip dependency caching

Reference Issues/PRs

What does this implement/fix? Explain your changes.

This PR adds further caching to the CI allowing all dependencies and wheels to be cache'd and restored locally greatly speeding up the CI.

One of the main bottle necks for the CI currently is installing dependencies and building wheels. Currently every time the CI is ran dependencies are installed from pip and built. In the 'perfect' scenario this is fine and installs and building of wheels takes around 5 minutes or less. However, due to external factors such as network issues, a bad CI node and other factors sometimes this install and building step can take upwards of 20 minutes. Normally in every CI run, one runner takes 15+ minutes to install and build the dependencies. As the CI is only as fast as its slowest runner this presents a bottleneck.

This PR caches the dependencies and the wheels built each time a PR is merged to main (if pyproject.toml is updated). As a result the CI can the restore the dependencies and wheel on all branches and runs skipping the download and build phase. This reduces the time to install down to around 3 minutes in the best case and in the worse case (due to bad runner) 7 minutes. This greatly speeds up the CI as the slowest runner should be much faster.

Does your contribution introduce a new dependency? If yes, which one?

Any other comments?

PR checklist

For all contributions

[x] I've added myself to the list of contributors. Alternatively, you can use the @all-contributors bot to do this for you.
[x] The PR title starts with either [ENH], [MNT], [DOC], [BUG], [REF], [DEP] or [GOV] indicating whether the PR topic is related to enhancement, maintenance, documentation, bugs, refactoring, deprecation or governance.

For new estimators and functions

[x] I've added the estimator to the online API documentation.
[x] (OPTIONAL) I've added myself as a __maintainer__ at the top of relevant files and want to be contacted regarding its maintenance. Unmaintained files may be removed. This is for the full file, and you should not add yourself if you are just making minor changes or do not want to help maintain its contents.

For developers with write access

[x] (OPTIONAL) I've updated aeon's CODEOWNERS to receive notifications about future changes to these files.

Mar 27 '24 17:03 chrisholder

Thank you for contributing to `aeon`

I have added the following labels to this PR based on the title: [ $\color{#EC843A}{\textsf{maintenance}}$ ]. I have added the following labels to this PR based on the changes made: [ $\color{#5209C9}{\textsf{distances}}$, $\color{#2C2F20}{\textsf{testing}}$ ]. Feel free to change these if they do not properly represent the PR.

The Checks tab will show the status of our automated tests. You can click on individual test runs in the tab or "Details" in the panel below to see more information if there is a failure.

If our pre-commit code quality check fails, any trivial fixes will automatically be pushed to your PR unless it is a draft.

Don't hesitate to ask questions on the aeon Slack channel if you have any.

Mar 27 '24 17:03 aeon-actions-bot[bot]

One key change worth documenting is in the cache script there is this step:

      - name: Install CPU version of pytorch on linux
        uses: nick-fields/retry@v3
        if: steps.cache.outputs.cache-hit != 'true' && runner.os == 'Linux'
        with:
          timeout_minutes: 30
          max_attempts: 3
          command: python -m pip install torch --index-url https://download.pytorch.org/whl/cpu

I found the default pip install torch to be multiple gbs in size because it installs all the GPU dependencies (which we dont use in the CI). I instead opt to only install the CPU dependencies for the cache which is only 200mbs of size. This allowed us to stay under the 10gb limit

Mar 27 '24 18:03 chrisholder

is this sped up?