Using "RUN --mount=type=cache" in rootless podman attempts to use FUSE

[rogu-beta]

Attempting to build a Dockerfile inside a CI pipeline fails when it contains instructions such as: RUN --mount=type=cache,target=/var/cache/apt,sharing=locked

The CI pipeline is running on rootless podman and thus buildah is executed with STORAGE_DRIVER: vfs and --isolation="chroot". Unfortunately, buildah appears to be unable to run the aforementioned instruction, insisting on using FUSE.

The error is as follows:

 mounting new container: mounting build container "b9d2683a6fefca311a22bb65b52f366debb0c828da51d2b01c8bf4577eae3fb4": creating overlay mount to /var/lib/containers/storage/overlay/26eb81fb7f8443c9c1a1d1964f5ec83b285031b92d1f5e698b41aeb783ad0470/merged, mount_data="lowerdir=/var/lib/containers/storage/overlay/l/4XYQU7AG33REVIGOPGZRZZGYND:/var/lib/containers/storage/overlay/l/QHP3KRPHZO3EZGG7E74FL4VSXE:/var/lib/containers/storage/overlay/l/GNHDIKHQ4X2N7PTYJMB6GNQD4A:/var/lib/containers/storage/overlay/l/TWEYBLT2SS6PYYNYTEZSRZWSHC:/var/lib/containers/storage/overlay/l/7GGKQEPI4ARZPK7UCQLQCN2IK3:/var/lib/containers/storage/overlay/l/QDW4Y4W52QXQ6OKHAZVGDAQAL4:/var/lib/containers/storage/overlay/l/KQ6EUUHLDSQ5LBRYF3DSEYMS7C:/var/lib/containers/storage/overlay/l/KFRJYCMICVN34IIE2YODP7V7NG,upperdir=/var/lib/containers/storage/overlay/26eb81fb7f8443c9c1a1d1964f5ec83b285031b92d1f5e698b41aeb783ad0470/diff,workdir=/var/lib/containers/storage/overlay/26eb81fb7f8443c9c1a1d1964f5ec83b285031b92d1f5e698b41aeb783ad0470/work,,volatile": using mount program /usr/bin/fuse-overlayfs: unknown argument ignored: lazytime
unknown argument ignored: 
fuse: device not found, try 'modprobe fuse' first
fuse-overlayfs: cannot mount: No such file or directory

If buildah is configured to use VFS and no usernamespacing, then I would have expected that RUN --mount=type=cache would also not attempt to use the overlay or namespacing either.

[nalind]

Can you confirm the output of buildah info in this setup? Which image was this using? The mountpoint which it was attempting to use was /var/lib/containers/storage/overlay/26eb81fb7f8443c9c1a1d1964f5ec83b285031b92d1f5e698b41aeb783ad0470/merged, which looks like an error being encountered while attempting to mount the working container's rootfs, rather than the cache's target location.

[rogu-beta]

The output is a follows, according to a colleague of mine:

{
    "host": {
        "CgroupVersion": "v2",
        "Distribution": {
            "distribution": "debian",
            "version": "12"
        },
        "MemFree": 4227371008,
        "MemTotal": 33654841344,
        "OCIRuntime": "crun",
        "SwapFree": 920440832,
        "SwapTotal": 1027600384,
        "arch": "amd64",
        "cpus": 8,
        "hostname": "runner-8xr2geijq-project-1376-concurrent-0",
        "kernel": "6.1.0-31-amd64",
        "os": "linux",
        "rootless": true,
        "uptime": "529h 41m 45.64s (Approximately 22.04 days)",
        "variant": ""
    },
    "store": {
        "ContainerStore": {
            "number": 0
        },
        "GraphDriverName": "overlay",
        "GraphOptions": null,
        "GraphRoot": "/var/lib/containers/storage",
        "GraphStatus": {
            "Backing Filesystem": "extfs",
            "Native Overlay Diff": "true",
            "Supports d_type": "true",
            "Using metacopy": "false"
        },
        "ImageStore": {
            "number": 0
        },
        "RunRoot": "/run/containers/storage"
    }
}

The container image buildah was being run in was the most recent node:22 image

[nalind]

Well, the "GraphDriverName": "overlay", suggests that something's gone awry setting $STORAGE_DRIVER for the container.

[rogu-beta]

@nalind The container in which buildah is being run can and should use overlay just fine, buildah however should not as it will otherwise will run into issues running nested, similar to nested containers. This setup works fine for building all sorts of Dockerfiles, until we ran across that case yesterday. I will double check with my colleague though whether the buildah info command was run with the same env vars enabled as the build job.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.