mount=type=cache more in-depth explanation?

[peci1]

trafficstars

Hi, I'm trying to use cache mounts to speed up things like apt install, pip install or ccache when rebuilding my containers.

For each of these, I prefix each RUN command using apt/pip/ccache... with things like:

RUN --mount=type=cache,target=/var/cache/apt,rw --mount=type=cache,target=/var/lib/apt,rw  \

That seems to work sometimes, but I found out I can't understand the cache invalidation rules. The cache just from time to time disappears and I have to download all the cached packages again. I made explicit effort to remove all apt clean commands from the dockerfile, so the cache is not deleted programatically. And it also happens to CCache, which is not autodeleted.

Could somebody please explain how does it work and when exactly can the cache be reused, when (if at all) is it discarded, where is it saved...? What is the exact effect of the id parameter?

I also found out that if I want to create a cache directory for a non-root user, I can create it with cache mount option uid=1000. But if I use uid=1000,gid=1000, the folder gets ownership root:root, which is really weird. Can it be connected with me not using id and setting some caches with uid=0 and some with uid=1000? Does this create some kind of conflict?

And is there actually a good reason for setting the mode to anything else than 777? If it's only used during build, I don't get why anybody would care about permissions...

Thank you for helping.

Maybe some of the answers could be added to https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md ?

[peci1]

And is it affected by the --no-cache build option?

[YesYouKenSpace]

If I may hijack this, I like to ask if RUN --mount=type=cache can use the cache image in --cache-from ?

[morlay]

confused for me too.

--cache-from and --cache-to do nothing for --mount=type=cache https://github.com/docker/buildx/issues/399

--mount=type=cache cached dir in buildkit container /var/lib/buildkit/ until gc cleanup.

[FernandoMiguel]

I have quite a few cache examples here https://github.com/FernandoMiguel/Buildkit

I suspect your issue is GC which by default is very small. also --no-cache will not use cache for that run

[morlay]

@FernandoMiguel use remote buildkit could resolve by increasing gc limits. i setup it for my private projects.

but some ci like Travis or GitHub Workflows

buildkit is always created. --mount=type=cache cached files in container, will be lost.

[FernandoMiguel]

Yes for any ephemeral host, cache is lost. That is expected and the correct behaviour.

You can use cache-from to pull layers from a image repo in that case

[morlay]

close https://github.com/docker/buildx/issues/399 and discussing here.

@FernandoMiguel

any suggestion in this case?. i need cache /go/pkg/mod for multi workflows in build stage.

but the --cache-to couldn't expose the cached files (/var/lib/buildkit) to host. and buildkit recreated when each workflow starts, all cached files in /var/lib/buildkit losts

buildx not provide way to mount host path for /var/lib/buildkit when create buildkit

[FernandoMiguel]

No idea what your Dockerfile looks like or what ci you are using That makes all the difference Some ci will allow you to cache host data

[morlay]

here example

workflow.yml https://github.com/querycap/istio/blob/master/.github/workflows/istio-pilot.yml Dockerfile https://github.com/querycap/istio/blob/master/build/istio/Dockerfile.pilot

and the logs https://pipelines.actions.githubusercontent.com/qcc22rpVKD7YzUzpABpz6Nbu6TDqQ0MFAEbSsnK9GDsCU4Hq5g/_apis/pipelines/1/runs/325/signedlogcontent/3?urlExpires=2020-09-24T13%3A32%3A15.1287086Z&urlSigningMethod=HMACV1&urlSignature=MDAD0FQhxCis3TFZLKqjsdmvB2m0iiCBWv6G9FyDOGs%3D

--cache-from --cache-to host path /tmp/.buildx/cache

--mount=type=cache not working

i understands how --mount=type=cache work.

i have no idea to cache files in buildkit container /var/lib/buildkit with docker buildx directly.

[FernandoMiguel]

Type cache won't obviously work on github actions since every run is done a new host. You can run your own host and control that, or hack a way to store the cache (github limits are too low for any real use)

[morlay]

@FernandoMiguel mount host path like /tmp/buildkit to buildkit to /var/lib/buildkit, then i add /tmp/buildkit to actions cache. could this be possible?

[FernandoMiguel]

What

[morlay]

docker buildx create --use --name localbuild
docker buildx inspect localbuild --bootstrap

# recreate buildkit with host path
docker rm -f buildx_buildkit_localbuild0
docker run --privileged -d --name=buildx_buildkit_localbuild0 -v=/tmp/buildkit:/var/lib/buildkit moby/buildkit:buildx-stable-1         

now /tmp/buildkit in host contains buildkit files /var/lib/buildkit i think i could cache this folder /tmp/buildkit

@FernandoMiguel it's not work with lots of permission error for whole path.

hope --cache-from --cache-to could support --mount=cache.

or add a option to path to assign the snapshots host path --mount=type=cache,path=/tmp/gomod,target=/go/pkg/mod

or replace the snapshot id of dirname with the id of --mount=type=cache,id=gomod,target=/go/pkg/mod /var/lib/buildkit/runc-overlayfs/snapshots/snapshots/2/go/pkg/mod /var/lib/buildkit/runc-overlayfs/snapshots/snapshots/gomod/go/pkg/mod

[FernandoMiguel]

/tmp is a special folder and may have weird permissions

[morlay]

github actions use a non-root user, but /var/lib/buildkit/runc-overlayfs only for root user.

and /var/lib/buildkit/runc-overlayfs always in changing. i give up to use this way.
need find other hacks.

[Aposhian]

Since this thread has been hijacked for a different issue, @peci1's questions are still unanswered. What exactly does id do? What are the cache invalidation rules?

[ringerc]

It's notable that the docs in https://docs.docker.com/engine/reference/commandline/buildx_build/ don't mention the cache, too.

[spacedub]

@Aposhian

I just spent some time testing cache sharing modes (private, locked, shared) in buildkitd/buildctl - I do assume the behavior is the same with buildx.

Here is my understanding:

• different ID or different mode -> different cache object - if you change the id (or the mode), you get a new empty cache mount
• same ID and same mode -> by default, same cache object (including across unrelated, different Dockerfiles), EXCEPT in the following circumstances:
  • you are building with --no-cache, in which case you get a new clean mount, that will then be used for that id moving forward
  • you are using mode private and there is already another build already running using that exact mount, in which case you also get a new one (that similarly will be used by subsequent builds using that mount ID) - that last part is especially confusing

Note that the cache object is the same even if you change the mount path... only the ID+mode matters...

So, same ID and mode, the cache object should not "disappear" on its own... ... that is, unless: a. you do a buildctl prune against your buildkitd, which destroys the cache objects b. or you run your build with --no-cache, in which case all cache objects used by that build will be reset (including for other concurrent builds that are still in-flight) c. or garbage collection decided to evict that cache entry d. or you are using a "private" mount and starting your build while another build is already accessing the mount e. OR... the apt-get configuration itself, inside your build is purposefully deleting the cache entries after a run

Furthermore, in the OP question, it looks to me like the sharing mode is shared (which is the default behavior). For use with apt, this is probably wrong. According to Docker documentation "apt needs exclusive access to its data" and their documentation suggests using locked instead https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/reference.md#run---mounttypecache

Of course, if you are running a lot (a few?) concurrent builds on that buildkitd, locked will definitely slow them all down and make you loose some / most of the caching benefits in the first place...

Whether or not locked (or private) is enough in the apt world... If you have multiple different apt versions in different images using the same mount (id+sharingmode) for your apt, you might be in for trouble...

So, given OP used "shared": f. maybe concurrent apt access to the cache mount makes apt drop all the data in some cases?

Furthermore, the OP mounts /var/cache/apt. But if you look at cat /etc/apt/apt.conf.d/docker-clean inside the Debian official image, it is clear that this actually prevents any caching of the packages. So, solely mounting into /var/cache/apt is useless.

Finally, mounting and reusing /var/lib/apt also seems useless to me. The cache benefit is about 3 seconds on Debian (3.7s on empty cold start, vs. 0.7s once /var/lib/apt is populated), in the rare case where the build instruction itself is not cached. And since you cannot trust the content of the cache folder... you cannot save yourself the apt-get update operation anyway in further instructions...

In a shell, if you want to use cache properly for apt:

• use at least locked, possibly private but do NOT expect it to be actually private to this specific image you are building
• use an ID that is truly unique to your Dockerfile or build process, unless you absolutely understand which other images you may be building that are using the same ID are not going to mess it up for your usage (different apt version for example)
• do not cache /var/lib/apt - the unlikely cache benefit does not make much / any sense - use a tmpfs instead, if the objective is to minimize the amount of stuff that gets baked into your final image
• do cache /var/cache/apt if you want (you do get the best bang for the buck here, by caching actual packages downloads), but be sure to ALSO configure apt to use it, as it is disabled in apt-conf.d in the official images (eg: that is option Dir::Cache)

Assuming you get the above right, the only remaining reasons for cache to disappear are a. b. or c from above (pretty much explicit prune or cache bust, or garbage collection).

Hope that helps.

[spacedub]

Thought I would clarify what happens when you do NOT use an id.

It will default to the target (eg: mount path). So, in OP case, the cache for apt is shared, meaning:

• it can be accessed concurrently by many different process
• any other mount binding into /var/cache/apt with the same sharing mode will reuse that SAME cache object

[peci1]

Thanks for the analysis.

But if you look at cat /etc/apt/apt.conf.d/docker-clean inside the Debian official image, it is clear that this actually prevents any caching of the packages.

I have this at the beginning of the dockerfile:

RUN sudo rm -f /etc/apt/apt.conf.d/docker-clean; echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' | sudo tee /etc/apt/apt.conf.d/keep-cache

So this should not interfere with the cache.

[peci1]

And I do not build concurrently, so private/shared is not a problem for me.

[Aposhian]

Thanks @spacedub!

That is really good to know that:

• cache mounts are identified by id and mode only (not uid/gid?)
• --no-cache creates a new cache mount that is used going forward (I had assumed it used no cache mount at all). That seems like something that would be good to document...

I think it would be interesting to experiment to see if sharing /var/cache/apt mounts between OS versions would cause problems. My first guess is not since the package downloads are specified by version and codename: for example: docker-ce_5%3a20.10.18~3-0~ubuntu-focal_amd64.deb

Why would you not cache /var/lib/apt, even if the gain is smaller relative to /var/cache/apt? Specifically /var/lib/apt/lists? Isn't that mainly caching work from apt-get update, fetching package lists from repositories? I personally like just making it a cache mount, and then I can skip the step of doing something like rm -rf /var/lib/apt/lists in my Dockerfile.

but be sure to ALSO configure apt to use it, as it is disabled in apt-conf.d in the official images (eg: that is option Dir::Cache)

I believe this is also circumvented by the one liner that @peci1 shared

[spacedub]

UID/GID I have not tested so I do not know (I will check later today) That being said I would be weary of using the same id and sharing mode with different uids in different places - seems quite error prone / confusing to me.

About the "lists" part, if your purpose is to avoid having to rm, a tmpfs should give you the same benefits without the downsides of locking / waiting. But then, if you are mostly building one thing at a time and mostly the same thing, "shared" is probably acceptable.

About the package cache - what happens if a different package with that same name has been downloaded in that location? Will apt verify the checksum of that (and then what happens next) or is apt verifying the checksum before, at download time? Will different Debian based distributions (or different versions of the same distribution) use different package names and versions for the same things, or can they conflict? Short of having clarity on these... I would be careful using the same share across unrelated Dockerfiles...

[fkhantsi]

hey, back to the first question, reusing cache among different ci/cd hosts.

I am using AWS CodeBuild to build an image, which allows me to store specific directories between builds, even though the host will go away. I just need to know which directory from the host I should persist.

Is it /var/lib/buildkit? is it /var/lib/docker/buildkit/cache.db as per #1474 ?

[spacedub]

To share cache between hosts, I would suggest using registry stored cache - see https://github.com/moby/buildkit#export-cache

Trying to share host folders used by buildkit between machines does not sound like a good idea...

[fkhantsi]

is this specifically the mount=type=run cache, or layer cache?

[spacedub]

Layer cache AFAIK. I would not use mount cache as a form of persistent storage... and I would not try to share them across hosts.

https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/reference.md#run---mounttypecache

Does spell it out:

Cache mounts should only be used for better performance. Your build should work with any contents of the cache directory as another build may overwrite the files or GC may clean it if more storage space is needed.

If you want something to persist, put it in a scratch image and push it - or leverage layer cache.

Hope that helps.

[fkhantsi]

Thanks, but this doesn't help me. I do want to use cache for better performance -- specifically for local maven repository.

Here is what I am trying to do: use a multi stage Dockerfile to first build my .war in a maven container, and then deploy it to a tomcat container. The build process should be identical on my dev machine, and in my CodeBuild CI/CD environment.

Without any optimization, I would be rebuilding all of the images, and downloading all of my maven dependencies on every build.

First optimization, which is not relevant to this discussion, is to publish the build image to a container registry, and build using cache-from. This will cache the layers, however if I change my pom.xml (maven's dependency file for those who don't know), the layer will be invalidated, and it will have to rebuild it, and redownload all the dependencies from maven central, instead of just one.

The solution for this is using --mount=type=cache target=/root/.m2 Unfortunately, since the host is ephemeral, the cache will be destroyed. I want to persist the cache between builds by saving it to s3, something that codebuild supports, by asking only which directory in the host to store between builds.

So which directory do I give it? /var/lib/buildkit? /var/lib/docker/buildkit/cache.db or something else?

[spacedub]

Thanks for clarifying. Why not use --mount=type=bind then (instead of --mount=type=cache)?

Pretty much gives you what you are asking for, and then you can just save that specific folder anyway you want.

[fkhantsi]

because it's for reading from host, not writing. Even if you make it read-write, the writes are discarded