systemd
Support a cache directory for use in prepare scripts
Currently there seems to be no good way to cache dependencies that are downloaded during a prepare script, e.g. by language-specific package managers like pip or npm.
I think it would be quite useful if we could specify a cache directory which persists across builds and is mounted in prepare scripts (and maybe build scripts, for use with WithNetwork=true) so that such package managers and similar tools can be pointed to said directory and avoid having to download the same dependencies on every build.
Use the image itself as the cache directory. Put stuff somewhere in /usr/xxx or so when the prepare script is executed with the "build" argument and you have your cache.
Maybe I'm misunderstanding but I don't see how that can persist a cache between builds, since the build image is built new every time. If I use, say, /usr/pip-cache then that directory will never exist when the prepare script runs, so the "cache" has to be populated every time.
I'm looking for something that, like mkosi's own distribution package cache, will let me avoid re-downloading all the build dependencies every time I build the image.
I could keep them in the source dir but I like to use BuildSourcesEphemeral=true, and ideally I would like to share this cache directory across multiple different mkosi images, just like how caches in e.g. ~/.cache can be used by all the pip/npm/whatever projects I work on outside of mkosi.
Good point, it probably makes sense to allow caching these downloads across image builds, we can probably make the package cache directory available to prepare scripts so extra stuff can be stored there as well.
So far I've usually used the BUILDDIR for these caching needs, e.g. setting in a build script
export GOCACHE="${BUILDDIR}/gocache"
export GOMODCACHE="${BUILDDIR}/gomodcache"
That works well enough. This doesn't work with prepare scripts, though, since we don't expose BUILDDIR to them. It does require additional setup, though, if you want to share the cache between different mkosi projects.
We discussed this a bit and we were thinking of adding explicit caching support for various programming language package managers instead. Would you be ok with that?
We discussed this a bit and we were thinking of adding explicit caching support for various programming language package managers instead. Would you be ok with that?
I think a more abstract solution might be better. E.g. I have used pip, poetry, git, npm, pnpm, cargo, and golang so far with mkosi and that is only for the ecosystems I use...
I also think something more generic would be best. Perhaps a setting for a cache dir that can either be set to a directory which is then used, or to a special value indicating that the global cache should be used (perhaps some subdirectory of ~/.cache/mkosi), or to a special value indicating that caching should be disabled. In the latter case, ideally inside the build container the cache directory and environment variable pointing to it should still exist but be ephemeral, so that scripts can just use the cache directory env var and don't have to worry about checking if it's set first.
Specific caching support for programming language package managers would of course be cool as well, but I think there are more such package managers than could reasonably be supported. There are also situations where dependencies need to be downloaded through other means than package managers, for example downloading said package managers or other build tools when they aren't packaged by the distro, CA certificates, or sources that aren't packaged but can't be vendored either) Most of these things aren't large enough that it's important to avoid downloading them again and again, but some might be and it would also help massively when working with an unreliable internet connection.