[NFS] failed to get xattr for /tmp/containerd-mount757554418/bin: operation not supported

[messense]

trafficstars

Hardware: Raspberry Pi 4 OS: Ubuntu 20.04 64-bit aarch64 Docker: Docker CE 20.10.7 linux/arm64 buildx: 0.5.1

#20 exporting to oci image format
#20 sha256:69a2560eef4d3ece902c3b5149d142e9bd132f25db0bc9e35b94201534c415d2
#20 exporting layers
#20 exporting layers 2.3s done
#20 ERROR: mount callback failed on /tmp/containerd-mount757554418: mount callback failed on /tmp/containerd-mount451692969: failed to write compressed diff: failed to create diff tar stream: failed to get xattr for /tmp/containerd-mount757554418/bin: operation not supported

https://github.com/messense/rust-musl-cross/runs/2867767893?check_suite_focus=true

[tonistiigi]

Could you confirm what backing filesystem+snapshotter is used in your system.

[messense]

Could you confirm what backing filesystem+snapshotter is used in your system.

It's NFS showing in df -T

root@raspberrypi:~# df -T
Filesystem                   Type     1K-blocks    Used Available Use% Mounted on
10.74.8.98:/exports/rpi-8-24 nfs      102687672 2344768  95083640   3% /
devtmpfs                     devtmpfs   3965824       0   3965824   0% /dev
tmpfs                        tmpfs      4000416       0   4000416   0% /dev/shm
tmpfs                        tmpfs       800084     588    799496   1% /run
tmpfs                        tmpfs         5120       0      5120   0% /run/lock
tmpfs                        tmpfs      4000416       0   4000416   0% /sys/fs/cgroup
tmpfs                        tmpfs       800080       0    800080   0% /run/user/0

docker info

Storage Driver: devicemapper
  Pool Name: docker-0:17-526256-pool
  Pool Blocksize: 65.54kB
  Base Device Size: 10.74GB
  Backing Filesystem: ext4
  Udev Sync Supported: true
  Data file: /dev/loop0
  Metadata file: /dev/loop1
  Data loop file: /var/lib/docker/devicemapper/devicemapper/data
  Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
  Data Space Used: 396.5MB
  Data Space Total: 107.4GB
  Data Space Available: 102.8GB
  Metadata Space Used: 17.72MB
  Metadata Space Total: 2.147GB
  Metadata Space Available: 2.13GB
  Thin Pool Minimum Free Space: 10.74GB
  Deferred Removal Enabled: true
  Deferred Deletion Enabled: true
  Deferred Deleted Device Count: 1
  Library Version: 1.02.167 (2019-11-30)

[cristiz1992]

I have the exact same issue. Is there a solution or a workaround ?

Thanks

[AkihiroSuda]

NFS is not supported. Mounting ext4/XFS/tmpfs on /tmp may work.

[messense]

I tried fuse_xattrs in the past but got other kind of errors.

[aggo15]

I encounter the same issue as well. I'm getting error message below when using buildx. The build process completed successfully and the error appear when attempting to push the image to registry. Same thing happen when I save the image to oci type.

#61 exporting to image
#61 exporting layers
#61 exporting layers 0.6s done
#61 ERROR: mount callback failed on /tmp/containerd-mount3022400872: mount callback failed on /tmp/containerd-mount2376661373: failed to write compressed diff: failed to create diff tar stream: failed to get xattr for /tmp/containerd-mount3022400872/bin: operation not supported
------
 > exporting to image:
------
error: failed to solve: mount callback failed on /tmp/containerd-mount3022400872: mount callback failed on /tmp/containerd-mount2376661373: failed to write compressed diff: failed to create diff tar stream: failed to get xattr for /tmp/containerd-mount3022400872/bin: operation not supported

The builder is using image moby/buildkit:buildx-stable-1 and the state files in /var/lib/buildkit are stored over NFS v4.2 share. Below is the volume I use in my docker-compose file.

volumes:
  nfsvolume:
    driver_opts:
      type: "nfs"
      o: "addr=<server_name>,nfsvers=4.2,port=2049,async"
      device: ":/"

Here is the mount info in the buildkit container

:/ on /var/lib/buildkit type nfs4 (rw,relatime,vers=4.2,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=<client_ip>,local_lock=none,addr=<server_ip>)

I also verified in buildkit container I can create xattr on a file in NFS share and show the attribute value.

/var/lib/buildkit # touch testfile
/var/lib/buildkit # setfattr -n user.comment -v "some comment" testfile
/var/lib/buildkit # getfattr -n user.comment testfile
# file: testfile
user.comment="some comment"

Despite the NFS share able to support extended attribute, the error message still persists. No issue encounter when I use normal volume. Is there any effort to support NFS share in the future for buildkit?

[PeterGrace]

@crazy-max sorry to necro an issue from a few years ago but I am also getting this error, but this time when building a multi-arch image using buildx kubernetes builders. Does the kubernetes buildx driver use nfs in the backend? I'm talking to two different builders, one running on amd64 and the other on arm64. I'm not sure how the buildx subsystem handles the ferrying of build data between the local system and the builder pods. (I am not on an NFS filesystem.)

 => ERROR exporting to image                                                                                                                                                                                0.3s
 => => exporting layers                                                                                                                                                                                     0.3s
------
 > exporting to image:
------
ERROR: failed to solve: mount callback failed on /tmp/containerd-mount573956862: mount callback failed on /tmp/containerd-mount2949133354: failed to write compressed diff: failed to create diff tar stream: failed to get xattr for /tmp/containerd-mount573956862/bin: operation not supported

[Arsobbiak]

There is a simple fix. Just set TMPDIR env var to a writable location