[WIP/TESTING] ci: Cache Docker images

[phlax]

This PR adds a cache job at the beginning of CI that remounts the local docker daemon to the faster scratch disk, downloads the build image into the daemon and then creates a gzipped archive of the docker data dir.

The subsequent jobs download the created tarball from azp cache into a tmpfs storage from where it is extracted into the custom docker data dir and docker is restarted with just the build image

The cache is keyed only on the build container sha tag, so the cache will expire very infrequently

Overall this saves ~average of 30s on all of the subsequent CI jobs (after adding a single additonal job of ~40s to check the cache - slightly longer in the rare event of the cache busting )

This will also alleviate the disk issue problems as the docker data dir is running from a separate disk and the existing /var/lib/docker is deleted

Using the scratch disk should be safe as it will mostly be accessed at the beginning when loading docker so should not contend with its use holding the swapfile

Using tmpfs to download the docker cache should also be safe as it is at the beginning of the job when the amount of mem is predictable, and it is immediately freed after access

Performance-wise the caveat is that it moves from the sometimes fast/slow docker download to the sometimes fast/slow cache - but i think on the whole the cache should be faster

There are a number of further optimizations that could be made as a follow up, such as putting the docker data dir on an LVM striped across both disks

Signed-off-by: Ryan Northey [email protected]

Commit Message: Additional Description: Risk Level: Testing: Docs Changes: Release Notes: Platform Specific Features: [Optional Runtime guard:] [Optional Fixes #Issue] [Optional Fixes commit #PR or SHA] [Optional Deprecated:] [Optional API Considerations:]

[phlax]

another possible optimization would be to use xz which has a multicore decompress mode for files that have specifically been created with a related compression mode

[github-actions[bot]]

This pull request has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in 7 days if no further activity occurs. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!