--cache-from does not work with ARG

[finkandreas]

I have the feeling that --cache-from does not work. Here are my steps to reproduce:

My Dockerfile:

FROM ubuntu:22.04

ARG NUM_PROCS=1

# update system
RUN echo "Running with $NUM_PROCS procs" && apt-get update && env DEBIAN_FRONTEND=noninteractive TZ=Europe/Zurich apt-get -yqq install --no-install-recommends build-essential cmake

Build image steps:

# work inside a container
docker run --privileged --rm -v Dockerfile:Dockerfile -it quay.io/buildah/stable:latest /bin/bash
buildah login my_registry.net
buildah bud -f Dockerfile --cache-from my_registry/path/to/podman_cache --cache_to my_registry/path/to/podman_cache --cache-ttl=600h --layers .

At this point I have built the image and during the build I was seeing log lines that told that buildah was pushing to cache, also I see cache layers in the repository.

Now redo the build steps:

# work inside a container
docker run --privileged --rm -v Dockerfile:Dockerfile -it quay.io/buildah/stable:latest /bin/bash
buildah login my_registry.net
buildah bud -f Dockerfile --cache-from my_registry/path/to/podman_cache --cache_to my_registry/path/to/podman_cache --cache-ttl=600h --layers .

I would assume that the second time I run the build it would reuse the cache for the apt-get update ... RUN step. But it is not considering the cache. I do not see any log output why it does not use any cache.

[ghost]

Agreed. --cache-from has a 100% cache miss depending on where build args are specified in the dockerfile. Seems to be a regression and a duplicate of https://github.com/containers/buildah/issues/2992

[omacranger]

Oddly enough we noticed the same thing this morning with 1.27.0, and 1.28.0 on separate hosts. We could not get remote caches to function with --cache-from before happening to try another dockerfile with no args present. Moving the args (that never changed) to be present just before use allowed previous layers to leverage the cache, but still not the layers using the args. I do not have the full dockerfile, but something like below was what we ran into:

Did not work (never used cached layers):

ARG ARG_VALUE

RUN npm install

RUN echo $ARG_VALUE 

Partially worked (cached the first layer, though not the second despite the arg value not changing):

RUN npm install

ARG ARG_VALUE

RUN echo $ARG_VALUE 

[ghost]

@rhatdan @flouthoc Bumping this to get your attention. The problem is pretty severe.

[flouthoc]

I'll take a look at this thanks for reporting.

[flouthoc]

Thanks PR: https://github.com/containers/buildah/pull/4326 should close this and newly added test should verify that we don't regress this in future again.

[ghost]

@flouthoc which version of buildah will this be available in? Looking at the changelogs of the current release 1.28.1 I don't see this in there.

[TomSweeneyRedHat]

@flyingsaucerer it's still upstream only at this point. v1.28.0 was created from the Main branch on October 3. The fix (#4326) was merged into the main branch on October 24. V1.28.1 was created today, but it was created from the v1.28.0 release branch, not the main branch, and the fix was not backported there. So this fix will be contained in v1.29.0 when it comes out. I suspect that will be happening in the next few weeks if not sooner.

[imraulkays]

@TomSweeneyRedHat Any news about 1.29 release?

[TomSweeneyRedHat]

@imraulkays, I was a bit off in my estimate for the release. The current plan is to create a Buildah v1.29 in time for release with Podman v4.4, which just released RC2. We are currently doing what we call the "vendor dance", getting all the versions bumped in the c/* projects that are vendored into Buildah and Podman, such as c/image, c/storage, and c/common. That vendor dance just started, once complete, Buildah will get the next bump to v1.29 before it is vendored into Podman v4.4.

Depending on our dance moves, we should have Buildah v1.29 on GitHub and in testing on Fedora later this week or next. I'd bet next Tues/Wed.