Build errors when copying files with extended attributes between stages

[legobeat]

Issue Description

When trying to perform rootless build this Containerfile, build fails with error copier: put: error setting extended attributes on "$file": setting value of extended attribute "user.containers.override_stat" on "$file": operation not permitted after updating podman to 5.0.3 (along other system packages - I can't say for sure it wasn't introduced by something else changing on the host).

The same Cointainerfile was building fine on the same system previously.

Steps to reproduce the issue

1. git clone -b 5.x --single-branch --depth 1 https://github.com/verdaccio/verdaccio && cd verdaccio
2. podman build --no-cache --pull --env NODE_OPTIONS='--no-network-family-autoselection --trace-warnings' -t verdaccio:v5.31.0

Describe the results you received

When trying to copy artifact between build stages, it exits with error:

[2/2] STEP 8/19: RUN mkdir -p /verdaccio/storage /verdaccio/plugins /verdaccio/conf
--> 21b7aa34c39a
[2/2] STEP 9/19: COPY --from=builder /opt/tarball .
Error: building at STEP "COPY --from=builder /opt/tarball .": storing "/home/user/.local/share/containers/storage/overlay/19b5c6717aa9f85e77bd76209236e19d1e184d18e38112a372a38a5f8e679021/merged/opt/tarball": error during bulk transfer for copier.request{Request:"PUT", Root:"/", preservedRoot:"/home/user/.local/share/containers/storage/overlay/78da49e58ed2bfd592595357508cbcbae7134063cb16d5863d703c15f9839611/merged/opt/verdaccio", rootPrefix:"/home/user/.local/share/containers/storage/overlay/78da49e58ed2bfd592595357508cbcbae7134063cb16d5863d703c15f9839611/merged/opt/verdaccio", Directory:"/", preservedDirectory:"/home/user/.local/share/containers/storage/overlay/78da49e58ed2bfd592595357508cbcbae7134063cb16d5863d703c15f9839611/merged/opt/verdaccio", Globs:[]string{}, preservedGlobs:[]string{}, StatOptions:copier.StatOptions{CheckForArchives:false, Excludes:[]string(nil)}, GetOptions:copier.GetOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), Excludes:[]string(nil), ExpandArchives:false, ChownDirs:(*idtools.IDPair)(nil), ChmodDirs:(*fs.FileMode)(nil), ChownFiles:(*idtools.IDPair)(nil), ChmodFiles:(*fs.FileMode)(nil), StripSetuidBit:false, StripSetgidBit:false, StripStickyBit:false, StripXattrs:false, KeepDirectoryNames:false, Rename:map[string]string(nil), NoDerefSymlinks:false, IgnoreUnreadable:false, NoCrossDevice:false}, PutOptions:copier.PutOptions{UIDMap:[]idtools.IDMap{}, GIDMap:[]idtools.IDMap{}, DefaultDirOwner:(*idtools.IDPair)(nil), DefaultDirMode:(*fs.FileMode)(nil), ChownDirs:(*idtools.IDPair)(nil), ChmodDirs:(*fs.FileMode)(nil), ChownFiles:(*idtools.IDPair)(nil), ChmodFiles:(*fs.FileMode)(nil), StripSetuidBit:false, StripSetgidBit:false, StripStickyBit:false, StripXattrs:false, IgnoreXattrErrors:false, IgnoreDevices:true, NoOverwriteDirNonDir:false, NoOverwriteNonDirDir:false, Rename:map[string]string(nil)}, MkdirOptions:copier.MkdirOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), ChownNew:(*idtools.IDPair)(nil), ChmodNew:(*fs.FileMode)(nil)}, RemoveOptions:copier.RemoveOptions{All:false}}: copier: put: error setting extended attributes on "/verdaccio.tgz": setting value of extended attribute "user.containers.override_stat" on "/verdaccio.tgz": operation not permitted

Describe the results you expected

Build succeeds

podman info output

host:
  arch: amd64
  buildahVersion: 1.35.4
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc40.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    version: "40"
  eventLogger: journald
  freeLocks: 2046
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.1.87
  linkmode: dynamic
  logDriver: journald
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.fc40.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-3.fc40.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.15-1.fc40.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.15
      commit: e6eacaf4034e84185fd8780ac9262bbf57082278
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240510.g7288448-1.fc40.x86_64
    version: |
      pasta 0^20240510.g7288448-1.fc40.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.force_mask: shared
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.13-1.fc40.x86_64
      Version: |-
        fusermount3 version: 3.16.2
        fuse-overlayfs: version 1.13-dev
        FUSE library version 3.16.2
        using FUSE kernel interface version 7.38
  graphRoot: /home/user/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 125
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/user/.local/share/containers/storage/volumes
version:
  APIVersion: 5.0.3
  GoVersion: go1.22.2
  Os: linux
  OsArch: linux/amd64
  Version: 5.0.3

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

$ sudo dnf list --installed | grep -E '^rpm-|ima[^a-z]'
ima-evm-utils.x86_64                        1.5-4.fc40                        @fedora
rpm-build-libs.x86_64                       4.19.1.1-1.fc40                   @fedora
rpm-libs.x86_64                             4.19.1.1-1.fc40                   @fedora
rpm-plugin-audit.x86_64                     4.19.1.1-1.fc40                   @fedora
rpm-plugin-selinux.x86_64                   4.19.1.1-1.fc40                   @fedora
rpm-plugin-systemd-inhibit.x86_64           4.19.1.1-1.fc40                   @fedora
rpm-sequoia.x86_64                          1.6.0-3.fc40                      @updates
rpm-sign-libs.x86_64                        4.19.1.1-1.fc40                   @fedora

Additional information

Similar and related issues

• https://github.com/containers/podman/issues/18543
• https://github.com/containers/podman/issues/18064

[rhatdan]

This looks like something in the underlying file system is blocking setting of user xattr on the files.

Please check if there is an SELinux issue? Try with setenforce 0 Try in rootfull mode. Most likely this is something about your homedir settings.

[rhatdan]

@giuseppe @nalind Thoughts?

[legobeat]

@rhatdan Thank you for following up! Does the below provide the info? Commands are executed from the same cwd where build is attempted, a subdirectory of user home (which is a bind-mount on the host itself, in case that could makes a difference)

$ sudo getenforce
Disabled

$ rm -f foo; echo test > foo

$ setfattr -n user.containers.override_stat -v bar foo

$ getfattr -n user.containers.override_stat foo

# file: foo
user.containers.override_stat="bar"

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[kvnull]

I just had the exact same issue trying to build the dockerfile from https://github.com/Mellanox/network-operator/ - I previously had to replace ~/.config/containers and one other place with symlinks to a local drive to fix xattr issues so it shouldn't be that.