actions
How can I debug slow caches?
Hi!
I noticed our github cache action has begun to take a considerably much longer time; what tools are there available to try and investigate this?
For instance here are two post actions for saving the cache:
https://github.com/openxla/stablehlo/actions/runs/7784115717/job/21224022761 Cache size 954MiB takes 1min32s to upload in GitHub Action
https://github.com/openxla/stablehlo/actions/runs/7997664838/job/21842544903 Cache size 1852MB takes 57m to upload GitHub action
There isn't much information to understand why the increase of cache size (2x) corresponds to such a dramatic increase in upload time.
Additionally, restoring the cache seems to have doubled;
https://github.com/openxla/stablehlo/actions/runs/7924329858/job/21635691286 (~20mins)
Received 1943706716 of 1943706716 (100.0%), 231.6 MBs/sec
Fri, 16 Feb 2024 00:51:50 GMT
/usr/bin/tar: ../../../.cache/bazel/_bazel_runner/install/14fb027596f626f2526df4873ea20b8b/platforms/os/BUILD: time stamp 2034-02-12 22:39:36 is 315352065.537809154 s in the future
Fri, 16 Feb 2024 01:09:04 GMT
Cache restored successfully
Fri, 16 Feb 2024 01:09:04 GMT
Cache restored from key: Linux-stablehlo_build_assets-d592c8ec8f7138dcbde6f0890d048e59cba95041-bazelbuild
1.8 GiB takes nearly 20 mins
https://github.com/openxla/stablehlo/actions/runs/7715868908/job/21031359892
Tue, 30 Jan 2024 18:38:03 GMT
/usr/bin/tar: ../../../.cache/bazel/_bazel_runner/install/f0478d8547df16461adab7eb1d945d57/xcode-locator: time stamp 2034-01-26 12:04:44 is 315250001.01019758 s in the future
Tue, 30 Jan 2024 18:38:03 GMT
Received 995097659 of 995097659 (100.0%), 315.7 MBs/sec
Tue, 30 Jan 2024 18:38:36 GMT
Cache restored successfully
Tue, 30 Jan 2024 18:38:37 GMT
Cache restored from key: Linux-stablehlo_build_assets-98509c7f9792c79b05a41b95c24607f6dd489c5a-bazelbuild
~1GiB takes 30 seconds?
Noticed a similar issue in https://github.com/golangci/golangci-lint-action/issues/297#issuecomment-2067601934 where caching <1MB takes > 2min:
2m 10s
Post job cleanup.
Omitting ~/.cache/go-build from cache directories
Omitting ~/go/pkg from cache directories
/usr/bin/tar --posix -cf cache.tzst --exclude cache.tzst -P -C /home/runner/work/evcc/evcc --files-from manifest.txt --use-compress-program zstdmt
Cache Size: ~0 MB (325880 B)
Cache saved successfully
Saved cache for golangci-lint from paths '/home/runner/.cache/golangci-lint' in 469ms
From logs it is not clear where that time is spent.
