buildah
buildah copied to clipboard
containers
RUN bind mount not cached when using extra build context
Issue Description
I have a Containerfile that includes a RUN with a bind mount to a file from a second build context specified using --build-context mycon=my_dir to the podman build command. No matter what I do this step is never reused/cached and is always rerun when building. My understanding is that this shouldn't be the case.
If I use the default build context and don't specify a from= in my RUN mount then it works fine. However even if I specify an additional build context that is the base context it fails (see below).
Steps to reproduce the issue
Steps to reproduce the issue
-
Create directory
test_mountingand change to it. -
Create
test_mounting.Containerfilewith content below. -
Run:
echo "TEST" >> test.txt -
Run
podman build --log-level info -t test-mounting:latest --build-context mycon=. -f test_mounting.Containerfile .
Containerfile
FROM centos:7 AS compilers
WORKDIR /work
# prove that some level of caching is working
RUN echo "Current directory: ${PWD}" && ls -lR .
#COPY --from=ccap test.txt new_test.txt
RUN --mount=type=bind,source=test.txt,target=test.txt,from=mycon stat -c '%a %u %U %g %G %W %X %Y %Z' test.txt
RUN ls -la /work && echo "DONE" >> a_file.txt
The non-mounted RUN calls are just to show that things are run and cached or not cached.
Describe the results you received
INFO[0000] podman filtering at log level info
INFO[0000] Using boltdb as database backend
INFO[0000] Setting parallel job count to 121
STEP 1/5: FROM centos:7 AS compilers
STEP 2/5: WORKDIR /work
--> Using cache 288eadf77cb754531263c358b1ca080d8a717c8baf6ef1093bd2edc8c89a4bd1
--> 288eadf77cb7
STEP 3/5: RUN echo "Current directory: ${PWD}" && ls -lR .
--> Using cache 1eb8be4dfbc26c8e66a5ed0ce37b1ae661f845c9febb6a057e9ba507d86d60a0
--> 1eb8be4dfbc2
STEP 4/5: RUN --mount=type=bind,source=test.txt,target=test.txt,from=mycon stat -c '%a %u %U %g %G %W %X %Y %Z' test.txt
644 0 root 0 root 0 1717293066 1717293066 1717293066
--> 339f29b8e127
STEP 5/5: RUN ls -la /work && echo "DONE" >> a_file.txt
total 0
drwxr-xr-x 1 root root 6 Jun 2 01:51 .
dr-xr-xr-x 1 root root 17 Jun 2 01:51 ..
COMMIT test-mounting:latest
--> 8c5f5e0f331e
Successfully tagged localhost/test-mounting:latest
8c5f5e0f331e982352fb257acccae7c6e1e426f911201a483ffa5efca95815e5
Describe the results you expected
I expected the RUN with the mount to be cached. In my real world case my extra build context is not the base context and I'm using --cache-to and --cache-from. The step right after this mount RUN is a lot of compiling so it is very noticeable when it isn't cached.
podman info output
host:
arch: amd64
buildahVersion: 1.33.7
cgroupControllers:
- memory
- pids
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: conmon-2.1.10-1.el9.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.1.10, commit: 3ea3d7f99779af0fcd69ec16c211a7dc3b4efb60'
cpuUtilization:
idlePercent: 99.69
systemPercent: 0.08
userPercent: 0.23
cpus: 40
databaseBackend: boltdb
distribution:
distribution: rocky
version: "9.4"
eventLogger: file
freeLocks: 2046
hostname: bumi.ssec.wisc.edu
idMappings:
gidmap:
- container_id: 0
host_id: 700
size: 1
- container_id: 1
host_id: 1672864
size: 65536
uidmap:
- container_id: 0
host_id: 53807
size: 1
- container_id: 1
host_id: 1672864
size: 65536
kernel: 5.14.0-284.11.1.el9_2.x86_64
linkmode: dynamic
logDriver: k8s-file
memFree: 12738236416
memTotal: 135071936512
networkBackend: netavark
networkBackendInfo:
backend: netavark
dns:
package: aardvark-dns-1.10.0-3.el9_4.x86_64
path: /usr/libexec/podman/aardvark-dns
version: aardvark-dns 1.10.0
package: netavark-1.10.3-1.el9.x86_64
path: /usr/libexec/podman/netavark
version: netavark 1.10.3
ociRuntime:
name: crun
package: crun-1.14.3-1.el9.x86_64
path: /usr/bin/crun
version: |-
crun version 1.14.3
commit: 1961d211ba98f532ea52d2e80f4c20359f241a98
rundir: /run/user/53807/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
pasta:
executable: ""
package: ""
version: ""
remoteSocket:
exists: false
path: /run/user/53807/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns-1.2.3-1.el9.x86_64
version: |-
slirp4netns version 1.2.3
commit: c22fde291bb35b354e6ca44d13be181c76a0a432
libslirp: 4.4.0
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.2
swapFree: 4287033344
swapTotal: 8002727936
uptime: 5394h 38m 55.00s (Approximately 224.75 days)
variant: ""
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
- ipvlan
volume:
- local
registries:
search:
- registry.access.redhat.com
- registry.redhat.io
- docker.io
store:
configFile: /home/davidh/.config/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions: {}
graphRoot: /home/davidh/.local/share/containers/storage
graphRootAllocated: 214006284288
graphRootUsed: 176895717376
graphStatus:
Backing Filesystem: xfs
Native Overlay Diff: "true"
Supports d_type: "true"
Supports shifting: "false"
Supports volatile: "true"
Using metacopy: "false"
imageCopyTmpDir: /var/tmp
imageStore:
number: 36
runRoot: /run/user/53807/containers
transientStore: false
volumePath: /home/davidh/.local/share/containers/storage/volumes
version:
APIVersion: 4.9.4-rhel
Built: 1714526144
BuiltTime: Wed May 1 01:15:44 2024
GitCommit: ""
GoVersion: go1.21.9 (Red Hat 1.21.9-2.el9_4)
Os: linux
OsArch: linux/amd64
Version: 4.9.4-rhel
Podman in a container
No
Privileged Or Rootless
Rootless
Upstream Latest Release
No
Additional environment details
$ uname -a
Linux bumi.ssec.wisc.edu 5.14.0-284.11.1.el9_2.x86_64 containers/podman#1 SMP PREEMPT_DYNAMIC Tue May 9 17:09:15 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
$ rpm -q podman
podman-4.9.4-3.el9_4.x86_64
Additional information
I originally noticed this in a local gitlab CI runner, but narrowed it down to the above example.
Side question: Does file modification time invalidate the layer cache for a COPY and/or bind mount?
I'm unfamiliar with this code base, but if this is expected to work but no one has the time to debug it I'd be grateful if someone could point me to a good starting location in the code. If I find the time maybe I can at least shorten the overall time it takes for this ticket to get resolved.
If it works with a docker build then it should work with buildah. Most likely there is no one able to work on this now.
Hm I thought I had tested this with docker, but it looks like you're right @rhatdan. This behaves the same with docker:
$ docker build -t test-mounting:latest --build-context mycon=. -f test_mounting.Containerfile .
[+] Building 0.9s (11/11) FINISHED docker:default
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build definition from test_mounting.Containerfile 0.0s
=> => transferring dockerfile: 395B 0.0s
=> [internal] load metadata for docker.io/library/centos:7 0.0s
=> [context mycon] load .dockerignore 0.0s
=> => transferring mycon: 2B 0.0s
=> [compilers 1/5] FROM docker.io/library/centos:7 0.0s
=> [context mycon] load from client 0.0s
=> => transferring mycon: 437B 0.0s
=> CACHED [compilers 2/5] WORKDIR /work 0.0s
=> CACHED [compilers 3/5] RUN echo "Current directory: ${PWD}" && ls -lR . 0.0s
=> [compilers 4/5] RUN --mount=type=bind,source=test.txt,target=test.txt,from=mycon stat -c '%a %u %U %g %G %W %X %Y %Z' test.txt 0.3s
=> [compilers 5/5] RUN ls -la /work && echo "DONE" >> a_file.txt 0.5s
=> exporting to image 0.1s
=> => exporting layers 0.1s
=> => writing image sha256:6824f59b7c8a214595432afecfcde008e1ebf0bd715eb58814bf3a385f238bfd 0.0s
=> => naming to docker.io/library/test-mounting:latest
Note no CACHE line for the RUN --mount. If I remove the Containerfile's from=mycon portion and use the base context then I see the CACHE line:
=> CACHED [compilers 2/5] WORKDIR /work 0.0s
=> CACHED [compilers 3/5] RUN echo "Current directory: ${PWD}" && ls -lR . 0.0s
=> CACHED [compilers 4/5] RUN --mount=type=bind,source=test.txt,target=test.txt stat -c '%a %u %U %g %G %W %X %Y %Z' test.txt 0.0s
=> CACHED [compilers 5/5] RUN ls -la /work && echo "DONE" >> a_file.txt
So in that case...I guess I file this with docker somewhere?
EDIT: I found an old terminal session output where in my real world use case the docker output did cache the RUN --mount with the extra context. Now I don't know what's going on.
EDIT 2: Looks like docker caches it if the extra context isn't the current directory (or maybe the same as the base context).
EDIT 3: Podman doesn't seem to cache in any of these situations.
A friendly reminder that this issue had no activity for 30 days.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
I confirm the issue. I have a multistage build that uses RUN with bind mounts. With Docker it caches perfectly but with podman there is no way to make layer caching work.
