devcontainer with features - podman error relabel

[cedric-orange]

• VSCode Version: 1.96.1
• Local OS Version: Ubuntu 24.04.1 LTS
• Remote OS Version: [email protected] ms-vscode-remote.remote-containers-0.394.0
• Logs: Destination:/tmp/build-features-src/hello_0 Device:bind Flags:20481 ClearedFlags:0 PropagationFlags:[262144] Data:z Relabel: RecAttr:<nil> Extensions:0 IDMapping:<nil>}: bind mounts cannot have any filesystem-specific options applied"

Steps to Reproduce:

1.Use this devcontainer.json:

{
    "image": "mcr.microsoft.com/devcontainers/base:ubuntu",
    "features": {
        "ghcr.io/devcontainers/feature-starter/hello:1": {
            "greeting": "Hello"
        }
    }
}

2. Launch devcontainer build

$ devcontainer build
[7 ms] @devcontainers/cli 0.72.0. Node.js v20.18.1. linux 6.8.0-40-generic x64.
[4389 ms] Resolving Feature dependencies for 'ghcr.io/devcontainers/feature-starter/hello:1'...
[5848 ms] Files to omit: ''
[6367 ms] Files to omit: ''
[6380 ms] Start: Run: podman buildx build --load --build-context dev_containers_feature_content_source=/tmp/user/1000/devcontainercli-wgwb8517/container-features/0.72.0-1734645749964 --build-arg _DEV_CONTAINERS_BASE_IMAGE=mcr.microsoft.com/devcontainers/base:ubuntu --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp --target dev_containers_target_stage -f /tmp/user/1000/devcontainercli-wgwb8517/container-features/0.72.0-1734645749964/Dockerfile.extended -t vsc-methone-image-e6feb3b9a9328819577c3b28f7f1b0aecb646d4f17f7eedc9e5f8b4c85625497-features /tmp/user/1000/devcontainercli-wgwb8517/empty-folder
[1/2] STEP 1/4: FROM mcr.microsoft.com/devcontainers/base:ubuntu AS dev_containers_feature_content_normalize
Trying to pull mcr.microsoft.com/devcontainers/base:ubuntu...
Getting image source signatures
Copying blob ecf676af4420 skipped: already exists  
Copying blob cdba1ca17c41 skipped: already exists  
Copying blob 228b6f149bcd skipped: already exists  
Copying blob 6414378b6477 skipped: already exists  
Copying blob 4f4fb700ef54 skipped: already exists  
Copying blob 87c3881f12ec skipped: already exists  
Copying blob 43d4049c40f8 skipped: already exists  
Copying blob 1b35e41fb030 skipped: already exists  
Copying blob 8284ddf57c03 skipped: already exists  
Copying config 3620e3a7a8 done   | 
Writing manifest to image destination
[1/2] STEP 2/4: USER root
--> Using cache 3e70e09371b632e39c7bace4ad34034e0b232d09f7fb4d61df077265934f19eb
--> 3e70e09371b6
[1/2] STEP 3/4: COPY --from=dev_containers_feature_content_source devcontainer-features.builtin.env /tmp/build-features/
--> Using cache f48fc03b0548187ada6ee42d7dc7cb825cd94bae50350eced178e5a427733cbf
--> f48fc03b0548
[1/2] STEP 4/4: RUN chmod -R 0755 /tmp/build-features/
--> Using cache 6bd38451bfcb7689dfef5619546270333a55cdb8c739daa5b8372ceaca2d1101
--> 6bd38451bfcb
[2/2] STEP 1/9: FROM mcr.microsoft.com/devcontainers/base:ubuntu AS dev_containers_target_stage
[2/2] STEP 2/9: USER root
--> Using cache 3e70e09371b632e39c7bace4ad34034e0b232d09f7fb4d61df077265934f19eb
--> 3e70e09371b6
[2/2] STEP 3/9: RUN mkdir -p /tmp/dev-container-features
--> Using cache 71604c44effead2b577291005635d4ced15def571d438c21182c81671bceff8c
--> 71604c44effe
[2/2] STEP 4/9: COPY --from=dev_containers_feature_content_normalize /tmp/build-features/ /tmp/dev-container-features
--> Using cache 2315b86a36aba4322db55657e2666342cbf389d0cc6d1b97a41e54821f9ed7ea
--> 2315b86a36ab
[2/2] STEP 5/9: RUN echo "_CONTAINER_USER_HOME=$( (command -v getent >/dev/null 2>&1 && getent passwd 'root' || grep -E '^root|^[^:]*:[^:]*:root:' /etc/passwd || true) | cut -d: -f6)" >> /tmp/dev-container-features/devcontainer-features.builtin.env && echo "_REMOTE_USER_HOME=$( (command -v getent >/dev/null 2>&1 && getent passwd 'vscode' || grep -E '^vscode|^[^:]*:[^:]*:vscode:' /etc/passwd || true) | cut -d: -f6)" >> /tmp/dev-container-features/devcontainer-features.builtin.env
--> Using cache 07be7c67bf97b0ef49cca213721086f3681a39dab261440a06409653ab94b3ea
--> 07be7c67bf97
[2/2] STEP 6/9: RUN --mount=type=bind,from=dev_containers_feature_content_source,source=hello_0,target=/tmp/build-features-src/hello_0,z     cp -ar /tmp/build-features-src/hello_0 /tmp/dev-container-features  && chmod -R 0755 /tmp/dev-container-features/hello_0  && cd /tmp/dev-container-features/hello_0  && chmod +x ./devcontainer-features-install.sh  && ./devcontainer-features-install.sh  && rm -rf /tmp/dev-container-features/hello_0
error running container: from /usr/bin/runc creating container for [/bin/sh -c cp -ar /tmp/build-features-src/hello_0 /tmp/dev-container-features  && chmod -R 0755 /tmp/dev-container-features/hello_0  && cd /tmp/dev-container-features/hello_0  && chmod +x ./devcontainer-features-install.sh  && ./devcontainer-features-install.sh  && rm -rf /tmp/dev-container-features/hello_0]: time="2024-12-19T23:02:33+01:00" level=error msg="runc create failed: invalid mount &{Source:/tmp/user/1000/buildah4021412995/mnt/buildah-bind-target-11 Destination:/tmp/build-features-src/hello_0 Device:bind Flags:20481 ClearedFlags:0 PropagationFlags:[262144] Data:z Relabel: RecAttr:<nil> Extensions:0 IDMapping:<nil>}: bind mounts cannot have any filesystem-specific options applied"
: exit status 1
ERRO[0001] did not get container create message from subprocess: EOF 
Error: building at STEP "RUN --mount=type=bind,from=dev_containers_feature_content_source,source=hello_0,target=/tmp/build-features-src/hello_0,z cp -ar /tmp/build-features-src/hello_0 /tmp/dev-container-features  && chmod -R 0755 /tmp/dev-container-features/hello_0  && cd /tmp/dev-container-features/hello_0  && chmod +x ./devcontainer-features-install.sh  && ./devcontainer-features-install.sh  && rm -rf /tmp/dev-container-features/hello_0": while running runtime: exit status 1

About my podman configuration

$ podman info
host:
  arch: amd64
  buildahVersion: 1.33.7
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.10+ds1-1build2_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: unknown'
  cpuUtilization:
    idlePercent: 94.44
    systemPercent: 1.16
    userPercent: 4.39
  cpus: 8
  databaseBackend: sqlite
  distribution:
    codename: noble
    distribution: ubuntu
    version: "24.04"
  eventLogger: journald
  freeLocks: 2047
  hostname: yd-5cg2303bft
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 6.8.0-40-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 15536173056
  memTotal: 33323937792
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.4.0-5_amd64
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.4.0
    package: netavark_1.4.0-4_amd64
    path: /usr/lib/podman/netavark
    version: netavark 1.4.0
  ociRuntime:
    name: runc
    package: containerd.io_1.7.24-1_amd64
    path: /usr/bin/runc
    version: |-
      runc version 1.2.2
      commit: v1.2.2-0-g7cb3632
      spec: 1.2.0
      go: go1.22.9
      libseccomp: 2.5.5
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt_0.0~git20240220.1e6f92b-1_amd64
    version: |
      pasta unknown version
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.1-1build2_amd64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 2051010560
  swapTotal: 2051010560
  uptime: 2h 27m 46.00s (Approximately 0.08 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/wgwb8517/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/wgwb8517/.local/share/containers/storage
  graphRootAllocated: 498589663232
  graphRootUsed: 133656887296
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /tmp/user/1000
  imageStore:
    number: 121
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/wgwb8517/.local/share/containers/storage/volumes
version:
  APIVersion: 4.9.3
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.22.2
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.3

[cedric-orange]

If I edit generated Dockerfile container-features/0.72.0-1734645749964/Dockerfile.extended

And replace line:

RUN --mount=type=bind,from=dev_containers_feature_content_source,source=hello_0,target=/tmp/build-features-src/hello_0,z \

by:

RUN --mount=type=bind,from=dev_containers_feature_content_source,source=hello_0,target=/tmp/build-features-src/hello_0 \

And relaunch manualy: podman buildx build --load --build-context dev_containers_feature_content_source=/tmp/user/1000/devcontainercli-wgwb8517/container-features/0.72.0-1734645749964 --build-arg _DEV_CONTAINERS_BASE_IMAGE=mcr.microsoft.com/devcontainers/base:ubuntu --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp --target dev_containers_target_stage -f /tmp/user/1000/devcontainercli-wgwb8517/container-features/0.72.0-1734645749964/Dockerfile.extended -t vsc-methone-image-e6feb3b9a9328819577c3b28f7f1b0aecb646d4f17f7eedc9e5f8b4c85625497-features /tmp/user/1000/devcontainercli-wgwb8517/empty-folder

Command success.

[chrmarti]

We added the z flag for Podman in https://github.com/devcontainers/cli/issues/548. Any idea why it does not work in your case?

[cedric-orange]

I have more information:

It's containerd update which breaks this containerd.io:amd64 (1.7.23-1, 1.7.24-1)

When I downgrade to containerd 1.7.23, it's ok for me.

[kahrpatrick]

We added the z flag for Podman in devcontainers/cli#548. Any idea why it does not work in your case?

I am facing the same problem when using Podman. When looking at the Podman documentation, the error message actually makes sense: the z flag is not a valid option for --mount and is only supported by --volume. This is consistent with how it works in Docker. Did this change since you added the flag?

Removing the z flag works for me as I am not in an SELinux context.

[kahrpatrick]

We added the z flag for Podman in devcontainers/cli#548. Any idea why it does not work in your case?

I am facing the same problem when using Podman. When looking at the Podman documentation, the error message actually makes sense: the z flag is not a valid option for --mount and is only supported by --volume. This is consistent with how it works in Docker. Did this change since you added the flag?

Removing the z flag works for me as I am not in an SELinux context.

I can confirm that the error does not occur on a different system with the same Podman version where SELinux is enabled.

[chrmarti]

@l0rd Do you know if --security-opt label=disable has better compatibility? Is z failing on all systems without SELinux?

[l0rd]

These are different options: --security-opt label=disable disables the SELinux labels check, whereas z relabel every file object in the volume and can have undesired side effects.

There is no option z for --mount. The option z of --volume corresponds to the option relable=share of --mount. The example provided in the documentation:

--mount=type=bind,src=/path/on/host,dst=/path/in/container,relabel=shared

[chrmarti]

Thanks, I also see podman build doesn't support --security-opt label=disable. It appears that z was supported in earlier versions of buildah. Is relabel=share the right approach for these folders that we use to bring install scripts into the build? I think the scripts might write to these folders, so I'm not sure we could make them read-only.

[l0rd]

The --security-opt=label=disable should work for podman build too.

Anyway I suppose that in the build scenario you control the mounted folders (i.e. these are temporary folders that you manage, not users). So using the relabel option z is a valid option too as there is no risk to update labels of system files.

[chrmarti]

The relabel option made it into buildah 1.31.0 (https://github.com/containers/buildah/pull/4705) in 2023. --security-opt=label=disable might have been around for longer, but I couldn't find documentation on that. Any recommendation on what would be best? We could parse out the buildah version if that's necessary.

[l0rd]

--security-opt=label=disable makes podman behave like docker, which should be easier to maintain. Inspecting the git history, it looks like disable was there from the very beginning of ~Podman~ Buildah.

[ryozi-tn]

I was also experiencing the same issue with VSCode + WSL (Ubuntu 24.04) + VSCode Extension: Dev Containers (0.417.0) + podman environment. Let me share what I found:

• In devcontainers/cli#696, when using podman, the "bind" option now passes "z"
• In opencontainers/runc#3990, from runc v1.2.0-rc.1 onwards, passing "z" in "bind" options causes an error
• containerd 1.7.23 was using runc v1.1.14. containerd 1.7.24 was using runc v1.2.2 (which is why this doesn't occur in containerd 1.7.23)
• As a test, when I downgraded only runc to v1.1.15, it worked in my environment

I'm not very familiar with the details, but if we respect runc's behavior, I feel we should temporarily not add "bind" option.

[chrmarti]

I see it is still working with crun. I will look into using --security-opt=label=disable for better compatibility.

[chrmarti]

Fixed in Dev Containers 0.424.0-pre-release. Let me know if this works for you, thanks!

[ryozi-tn]

I upgraded to devcontainer v0.424.0 in VSCode's Extension and now it works on runc v1.2.5 in my environment!