buildkit
buildkit copied to clipboard
moby
Allow controlling cache mounts storage location
related https://github.com/moby/moby/issues/14080
Allowing exporting contents of the type=cache mounts have been asked many times in different repositories and slack.
https://github.com/moby/buildkit/issues/1474 https://github.com/docker/buildx/issues/244
Regular remote instruction cache does not work for cache mounts that are not tracked by cache key and are just a location on disk that can be shared by multiple builds.
Currently, the best approach to maintain this cache between nodes is to do it as part of the build. https://github.com/docker/buildx/issues/244#issuecomment-602750160
I don't think we should try to combine cache mounts with the remote cache backends. Usually, cache mounts are for throwaway cache and restoring it would take a similar time to just recreating it.
What we could do is to allow users to control where the cache location is on disk, in case it is not on top of the snapshots.
We can introduce a cache mount backend concept behind a go interface that different implementation can implement.
Eg. for a Dockerfile like
RUN --mount=type=cache,target=/root/.cache,id=gocache go build ...
you could invoke a build with
docker build --cache-mount id=gocache,type=volume,volume=myvolume .
In that case, the cache could use a Docker volume as a backend. I guess good drivers would be volume in Docker and bind mount from the host for non-docker. If no --cache-mount is specified, the usual snapshotter-based location is used.
From the security perspective, BuildKit API is considered secure by default for the host, so I think this would require daemon side configuration to enable what paths can be bound.
Another complexity is buildx container driver as we can't easily add new mounts to a running container atm. Possible solutions are to force these paths to be set on buildx create or do some hackery with mount propagation.
Just to clarify whether the scope of this proposal covers a use case I have.
Go has content-addressed build and module caches defined via go env GOCACHE and, as of 1.15, go env GOMODCACHE (which was (go env GOPATH)[0]/pkg/mod in previous go versions).
Have read the documentation at https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md, it does not appear possible to delegate a cache to a host directory. Therefore a buildkit Go build and module cache will likely largely duplicate the build and module caches that exists on the host.
However, this proposal seems to be heading in the direction of sharing such a cache:
In that case, the cache could use a Docker volume as a backend
Under this proposal, can I confirm, would it be possible to delegate these caches to host directories?
I note a couple of requirements:
- in the case of Go, the location of these directories is not guaranteed to be given by an environment variable, rather the output of
go env GOCACHEandgo env GOMODCACHEis definitive. Whilst not the end of the world if environment variables were the only way of passing values, support the output ofgo env GOCACHEandgo env GOMODCACHEwould be even better - the UID and GID of writes to the cache should default to be that of the caller
Apologies if this covers old ground (covered elsewhere); I'm rather new to the buildkit approach but happened upon this issue and it sounded like exactly the thing I was after.
Many thanks
hope this feature could land asap.
it will be useful for ci caching. https://github.com/moby/buildkit/issues/1673#issuecomment-698348524
for host directories, could be hacking by
# ensure host path
$ mkdir -p /tmp/gocache
# create volume
$ docker volume create --driver local \
--opt type=none \
--opt device=/tmp/gocache \
--opt o=bind \
myvolume
$ docker run -it --volume myvolume:/go/pkg/mod busybox touch /go/pkg/mod/test.txt
# test.txt will be created under host dir /tmp/gocache/
# maybe work
$ docker buildx build --cache-mount id=gocache,type=volume,volume=myvolume .
@tonistiigi should we control the mount target too?
--cache-mount id=gocache,type=volume,volume=myvolume,target=/go/pkg/mod
Does the proposed solution cater to a scenario of a buildserver which uses docker-in-docker? I'm not sure tbh.
This would be brilliant for build systems like Gradle and Maven building on e.g. GitHub Actions.
They typically download all their dependencies to a cache dir. It's hard to benefit from layer caching - dependencies can be expressed in multiple files in a nested folder structure, so to avoid a maintenance nightmare it's generally necessary for the Dockerfile to do a COPY . . before running the Gradle / Maven command that downloads the dependencies, which in turn means the layer cache is invalid nearly every time. Downloading the transitive dependencies is very chatty, can be hundreds of HTTP requests, so it's well worth turning into a single tar ball.
I really want to use the same Dockerfile to build locally and on CI, which I think means I don't want to use the strategy suggested in https://github.com/docker/buildx/issues/244#issuecomment-602750160 of loading & exporting the cache to named locations as commands in the Dockerfile - it might work in CI but would be much less efficient building locally, as well as adding a lot of noise to the Dockerfile.
I'm currently caching the whole of the /var/lib/docker dir (!) and restarting the docker service on the GitHub Action runner, which is also pretty slow and expensive, and generally not a great idea!
I'm guessing it wouldn't be a great place for a buildx AND go newbie to start contributing, though...
I would love to see
RUN --mount=type=cache,target=/root/.m2,id=mvncache mvn package
be exported as a own layer/blob in the --cache-to and --cache-from arguments like --cache-to type=registry,ref=myregistry/myapp:cache.
Perhaps even better if it got it's own argument like --cache-id id=mvncache,type=registry,ref=myregistry/dependency-cache:mvn and thus you can share maven dependencies between similar projects and avoiding downloading so much from the Internet.
I have hacked together a solution that seems to work for including cache mounts in the GitHub actions/cache action. It works by dumping the entire buildkit state dir to a tar file and archiving that in the cache, similar to the approach described here. I think this dump also includes the instruction cache, so this should not be archived separately if using the action.
Because this cache will grow on every run, we use docker buildx prune --force --keep-storage to remove everything but the cache before archiving. You will need to adjust the cache-max-size var to suit your needs, up to GitHub's limit of 5g. It's a dirty hack, and could probably be improved by exporting only the relevant cache mounts, but I am new to both buildkit and GitHub Actions so this is what I came up with. Comments and suggestions for different approaches are welcome.
Usually, cache mounts are for throwaway cache and restoring it would take a similar time to just recreating it.
@tonistiigi can you clarify what you mean by this? This sounds like it is only considering use cases such as caching dependency installation where downloading the dependencies from a repository would take the same time to download the cache itself. What about situations where regenerating the cache takes significantly longer than importing it (e.g. code building)? The desire is to re-use parts of the cache without erasing it entirely, something that the remote instruction cache does not support.
@chris13524 That's what "usually" means there, "not always". Even with code building cache you might have mixed results. Importing and exporting cache is not free. In addition you need something that can clean up the older cache from your mount or your cache just infinitely grows with every invocation. You can try this today by importing/exporting cache out of the builder into some persistent storage(or additional image) with just an additional built request. https://github.com/docker/buildx/issues/244#issuecomment-602750160
what is the progess of this feature?
I can really cry. Wasted many hours today to find a good workaround. This just sucks. Is this at least on some kind of roadmap?
@tonistiigi we have an increasing need to support this. Would you be open to receiving a PR to support it?
@tonistiigi we have an increasing need to support this. Would you be open to receiving a PR to support it?
There is the "help-wanted" label, so I guess someone with the required knowledge can open a PR.
It is very surprising that cache mounts do not work in tandem with the --cache-to and --cache-from flags. The lack of this feature makes cache mounts pretty useless in CI.
Creating a cache efficient docker build is tricky. "Hacks" are employed to create dependency only layers (i.e first copy package.json, npm install, and only than copy app files)
Having cache mounts work will make these hacks obsolete - docker building process will utilize the natural caching mechanism of the build systems, while keeping the cache available for the next CI run.
At the very least there should be a note about it in the docs, it would save hours of debugging.
The one use case I've found currently where cache mounts are actually helpful is where the same cache is used several times within a given docker build. E.g. a package manager like apk, apt, or dnf might be invoked several times. But for other things it's hard to see it being useful.
I have hacked together a solution that seems to work for including cache mounts in the GitHub
actions/cacheaction. It works by dumping the entire buildkit state dir to a tar file and archiving that in the cache, similar to the approach described here.
We're using self-hosted runners. How bad would it be to just make an AWS Elastic File System and mount it at /var/lib/buildkit?
I know NFS can be notoriously bad for system that require a lot of file system locking, but even though EFS speaks the NFS4 protocol, I assume AWS implements it better under the hood and maybe it doesn't have the same problems.
I have hacked together a solution that seems to work for including cache mounts in the GitHub
actions/cacheaction. It works by dumping the entire buildkit state dir to a tar file and archiving that in the cache, similar to the approach described here.We're using self-hosted runners. How bad would it be to just make an AWS Elastic File System and mount it at
/var/lib/buildkit?I know NFS can be notoriously bad for system that require a lot of file system locking, but even though EFS speaks the NFS4 protocol, I assume AWS implements it better under the hood and maybe it doesn't have the same problems.
Are you going to mount EFS into every buildkitd container? I thought that running multiple builders with shared storage is impossible. Even if you ensure that only one builder has access to the storage, I believe that this will degrade the performance of the build.
+1 on cache rebuild in particular when running cross-platform. emulated builds are computationally expensive such that even a python native build to .whl from a .tar.gz takes noticeable time and ffi builds (i.e. needing C compilation) take extremely long times.
This has been mentioned before as https://github.com/moby/buildkit/issues/1512#issuecomment-923300737 but we are feeling this to the tune of minutes added per (emulated) container.
Why are cache mounts a solution to the package.json problem you mentioned?
If you solely rely on cache mounts for, e.g., a npm install you won't
have to redownload the packages, but they'd have to be reinstalled even if
your package.json hasn't changed.
On Fri, Jul 22, 2022, 21:35 Alex Puschinsky @.***> wrote:
It is very surprising that cache mounts do not work in tandem with the --cache-to and --cache-from flags. The lack of this feature makes cache mounts pretty useless in CI.
Creating a cache efficient docker build is tricky. "Hacks" are employed to create dependency only layers (i.e first copy package.json, npm install, and only than copy app files)
Having cache mounts work will make these hacks obsolete - docker building process will utilize the natural caching mechanism of the build systems, while keeping the cache available for the next CI run.
At the very least there should be a note about it in the docs, it would save hours of debugging.
— Reply to this email directly, view it on GitHub https://github.com/moby/buildkit/issues/1512#issuecomment-1192878530, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAUCJ5DCHQ7CAH7PZFBEUVLVVLZWRANCNFSM4NMS4JEA . You are receiving this because you commented.Message ID: @.***>
To anyone looking for a workaround for initializing the mount-cache with your own external cache without build cache invalidation (in CI/CD, for example).
Prerequisites:
- your cache in the
var/cachedirectory
We need a multi-stage build:
# Copy the external cache from mount-bind into mount-cache.
# The mount-bind will invalidate the build cache but only in the current stage which is OK.
# No stages below depend on this stage, so the build cache of the below stages will not be invalidated.
FROM alpine:3.15 as cache-prepare
RUN --mount=type=cache,id=cache,dst=/var/cache \
--mount=type=bind,id=cache-ext,src=var/cache,dst=/var/cache-ext \
cp -R /var/cache-ext/* /var/cache/ || true # cp will fail if the cache-ext directory is empty
# Now we can use the pre-filled mount-cache
FROM **** as target
RUN --mount=type=cache,id=cache,dst=/var/cache \
<your commands here>
# Copy the cache from the mount-cache back to the build cache to be able to copy it back to the host
FROM alpine:3.15 as cache
# Add the invalidation mount-bind to not cache the cp command as changes in the mount-cache do not invalidate the build cache and you may get a cached and outdated result in the image.
RUN --mount=type=cache,id=cache,dst=/var/cache \
--mount=type=bind,id=invalidate,src=var/cache,dst=/tmp/dir \
cp -R /var/cache /tmp
Build script:
# Build number for the build cache invalidation if multiple executions on the same machine are made. You may omit it if only one build is made per machine instance.
build_num=$(cat .build-number || true)
build_num=${build_num:-0}
build_num=$((build_num+1))
echo $build_num > .build-number
# Make sure the mount-bind on the var/cache directory will invalidate the build cache
cp .build-number var/cache/.build-number
# Prepare the cache. Since no stages depend on it, it should be executed separately to initialize the mount-cache.
docker build --target cache-prepare --tag image:cache-prep .
# Main build
docker build --target target --tag image:target .
# Build a local image with the cache
docker build --target cache --tag image:cache .
# Copy the updated cache back to the host machine
docker rm -f cache-container || true # remove previous temporary container with the same name if any
docker create -ti --name cache-container image:cache # create the stopped temporary container on the image containing the cache.
rm -rf var/cache/* # cleanup the host cache directory before copying
docker cp -L cache-container:/tmp/cache - | tar -x -m -C var # copy files
docker rm -f cache-container # remove the temporary container
# Push the target image to your repository
docker push image:target
Now you have the updated cache in the var/cache directory and may store in a place of your choice and share it between instances. Any changes in the cache directory will not affect the build cache allowing to reuse of previously executed instructions efficiently. This may speedup the build very significantly.
This is a simplified example to demonstrate the principle. You will probably have to pull the previous Docker image and use it as the build cache with the --cache-from argument. You may also need the --build-arg BUILDKIT_INLINE_CACHE=1 to make Docker trust your local build cache between stages.
Images are built also locally (initial dev, debugging, faster iteration etc). In local builds I would really like to use the fact that I already have pretty much all dependencies cached locally (in say ~/.npm). With current buildkit cache I am forced to just build a copy of this sizable dir somewhere else. It works but why not allow me to say ... buildkit please take this cache from here please thanks.
Running everything in docker only to "solve" this by not using ~/.npm is not an option either.
Yet, I'd fear that being able to share a cache mount with the docker host would raise issues about the sharing=locked option?
Folks, I've captured @speller 's idea in a quick and dirty composite action at https://github.com/overmindtech/buildkit-cache-dance.
Good luck.
@speller - I'm looking through the Dockerfile and build script implementation you shared, and I just want to make sure I've wrapped my head around that entirely.
In essence, you:
- begin with a base cache located on the build host @
var/cache - pass that "source" cache into Docker
- allowing Docker to use the cache but also to mutate it as well by adding any new dependencies or requirements
- then you're essentially "rendering" that cache into its own docker image (by building & tagging that specific build stage)
- Finally, you're extracting the content of the "rendered" cache container OUT of the Docker image, and back onto the host so that the CI/CD pipeline (let's pretend it's a BitBucket pipeline that's configured to cache the
/var/cachedirectory) also caches that on its end?
Is that a correct walkthrough of the logic in the Dockerfile + script? (BTW, it seems really clever, and thanks for sharing it with everyone!)
Agreed, this is very clever... I had previously tried a similar method but hadn't thought of this technique to isolate the cache data for importing. Why is the layer cache busting file necessary though, wouldn't --no-cache work just as well? https://github.com/overmindtech/buildkit-cache-dance/blob/d79a86f66db5a59ac117cd185beebe8f4a7e8bfa/extract/action.yml#L16-L19
