buildkit-cache-dance
buildkit-cache-dance copied to clipboard
Large cache takes a long time to inject/extract
Mostly a discussion point, but I've noticed large caches take a very long time to inject/extract. In my case, I have a 467MB cache of npm modules. This is downloaded from github's cache into the workflow runner in 8s, and thus is often worth caching for GH workflows. However, the injection step takes 53s. It seems like a lot of this time is transferring context, supposedly 1.8GB worth (taking 30s). I'm not sure if this is because the context is no longer zipped like when GH downloads it, or if something else is causing the 500MB to triple in size. The extraction step takes 2m56s. Happy to provide any logs as needed.
@alex-statsig caching NPM modules usually negatively affects build performance, because you spend more time on extracting and saving cache back compared to just not using cache and simply downloading packages from npm registry each time. NPM modules consist from many files, that's why cache extraction/saving is expensive.