podman icon indicating copy to clipboard operation
podman copied to clipboard
verified publisher icon containers

unlinkat directory not empty on commit (rootless)

Open dg424 opened this issue 1 year ago • 6 comments

Issue Description

Getting the following error "randomly" when trying to build an image using the following command -> podman build --isolation chroot -t .

17:11:18 COMMIT foo:latest 17:11:18 --> 4434cee3fd5 17:11:18 Successfully tagged localhost/foo:latest 17:11:32 time="2024-06-14T21:11:30Z" level=error msg="error deleting build container \"dffdfc25f7f9f183eaca0c83ad95cd42daa5fbe0f33ec56cc8c525f3b0d5a98f\": 1 error occurred:\n\t* unlinkat /var/lib/containers/storage/vfs/dir/2157acd33ff63d42f27f2b14276c62bd0dea9d6d856849bad18da2220dfdf9e9: directory not empty\n\n\n" 17:11:32 Error: unlinkat /var/lib/containers/storage/vfs/dir/2157acd33ff63d42f27f2b14276c62bd0dea9d6d856849bad18da2220dfdf9e9: directory not empty

Any ideas/things to try etc ?

Steps to reproduce the issue

Unfortunately, the issue cannot be reproduced reliably and as stated in the description, it seems to occur randomly.

Describe the results you received

See description

Describe the results you expected

No error and a successful image build.

podman info output

bash-4.4# podman version
WARN[0000] Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user 
Client:       Podman Engine
Version:      4.4.1
API Version:  4.4.1
Go Version:   go1.19.10
Built:        Wed Oct  4 14:55:19 2023
OS/Arch:      linux/amd64

Podman in a container

Yes

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

Runs on an GCP compute engine instance

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

dg424 avatar Jun 17 '24 20:06 dg424

Additional environment details

Runs on an GCP compute engine instance

please provide more information on the environment. I see only one mapping is available. How was the user created? Is it a nested container?

podman info output

bash-4.4# podman version
WARN[0000] Using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding sub*ids if not using a network user 
Client:       Podman Engine
Version:      4.4.1
API Version:  4.4.1
Go Version:   go1.19.10
Built:        Wed Oct  4 14:55:19 2023
OS/Arch:      linux/amd64

podman version is not as helpful as the podman info output that is requested by the PR template. Can you please provide the podman info output?

giuseppe avatar Jun 18 '24 07:06 giuseppe

Hi @giuseppe,

Here is the output of podman info:

podman info --debug
time="2024-06-17T15:05:31Z" level=warning msg="\"/\" is not a shared mount, this could cause issues or missing mounts with rootless containers"
host:
  arch: amd64
  buildahVersion: 1.29.0
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.6-1.module+el8.8.0+1265+fa25dd7a.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: a88a21e8953a6243d5f369f61a342bcaf0630aa1'
  cpuUtilization:
    idlePercent: 84.2
    systemPercent: 2.37
    userPercent: 13.42
  cpus: 48
  distribution:
    distribution: '"rocky"'
    version: "8.8"
  eventLogger: file
  hostname: build-20240617150503150-l8s55-g8z2c
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 0
      size: 1
    - container_id: 1
      host_id: 1
      size: 4294967294
    uidmap:
    - container_id: 0
      host_id: 0
      size: 1
    - container_id: 1
      host_id: 1
      size: 4294967294
  kernel: 5.15.0-1050-gke
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 52360847360
  memTotal: 101331390464
  networkBackend: cni
  ociRuntime:
    name: runc
    package: runc-1.1.4-1.module+el8.8.0+1265+fa25dd7a.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.1.4
      spec: 1.0.2-dev
      go: go1.19.4
      libseccomp: 2.5.2
  os: linux
  remoteSocket:
    path: /run/user/0/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_SYS_CHROOT,CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-2.module+el8.8.0+1265+fa25dd7a.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 0
  swapTotal: 0
  uptime: 1439h 54m 32.00s (Approximately 59.96 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /root/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 3168432029696
  graphRootUsed: 1942340562944
  graphStatus: {}
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.4.1
  Built: 1696431319
  BuiltTime: Wed Oct  4 14:55:19 2023
  GitCommit: ""
  GoVersion: go1.19.10
  Os: linux
  OsArch: linux/amd64
  Version: 4.4.1

dg424 avatar Jun 18 '24 13:06 dg424

I see the message:

time="2024-06-17T15:05:31Z" level=warning msg="\"/\" is not a shared mount, this could cause issues or missing mounts with rootless containers"

In what environment are you running that command? Is it a nested container? Directly on the host?

Also this is the issue tracker for the upstream development, so please try with a newer version of Podman to see if the issue still persists

giuseppe avatar Jun 24 '24 11:06 giuseppe

We already tried with the latest version and still the same issue. In regards to the environment, this is a k8s pod running rootless docker daemon.

dg424 avatar Jun 24 '24 13:06 dg424

I've tried to reproduce a similar environment, running nested podman but I am not able to reproduce it yet.

Could you try to run podman inside podman (so no Docker involved) and see if that behaves in the same way for you? You can just use the podman image, e.g. podman run podman ...`

Could you share your Dockerfile?

giuseppe avatar Jun 25 '24 10:06 giuseppe

A friendly reminder that this issue had no activity for 30 days.

github-actions[bot] avatar Jul 26 '24 00:07 github-actions[bot]

Same issue seen here.

Environment:

  • ArchLinux
  • Podman 5.4.2
  • /dev/nvme0n1p2 on / type bcachefs (rw,noatime)
  • Linux xxx 6.14.7-5-cachyos #1 SMP PREEMPT_DYNAMIC Tue, 20 May 2025 05:45:19 +0000 x86_64 GNU/Linux
Successfully tagged ghcr.io/haikuports/haikuporter/buildmaster:1.3.0-3
ERRO[0331] error deleting build container "68c02ef39f252ad35e92c924cfe21321fa9a04dbb766fcd9b1f455b1730c367b": unlinkat /home/kallisti5/.local/share/containers/storage/overlay/6d76ecc42bf979027749efb68db2272dc824c9489752bb9a65d165644945583a/diff/usr/include/rdma: directory not empty
ERRO[0352] error deleting build container "9d6764de1f5192f4acaa15162e0e2acdc7a31f248a1244ebd5b1c9574df16228": unlinkat /home/kallisti5/.local/share/containers/storage/overlay/ebd43e399a9c48a2d83b543c81836495c446536674b8a3334af22c89f7347ff0/diff/tmp/buildtools/gcc/gmp/mpn/x86_64: directory not empty
Error: unlinkat /home/kallisti5/.local/share/containers/storage/overlay/ebd43e399a9c48a2d83b543c81836495c446536674b8a3334af22c89f7347ff0/diff/tmp/buildtools/gcc/gmp/mpn/x86_64: directory not empty
make: *** [Makefile:7: default] Error 125

$ podman info --debug
WARN[0000] Found incomplete layer "6d76ecc42bf979027749efb68db2272dc824c9489752bb9a65d165644945583a", deleting it
WARN[0000] Found incomplete layer "ebd43e399a9c48a2d83b543c81836495c446536674b8a3334af22c89f7347ff0", deleting it
host:
  arch: amd64
  buildahVersion: 1.39.4
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-1:2.1.13-1.1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: 82de887596ed8ee6d9b2ee85e4f167f307bb569b'
  cpuUtilization:
    idlePercent: 99.15
    systemPercent: 0.21
    userPercent: 0.65
  cpus: 32
  databaseBackend: sqlite
  distribution:
    distribution: cachyos
    version: unknown
  eventLogger: journald
  freeLocks: 2046
  hostname: eris
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.14.7-5-cachyos
  linkmode: dynamic
  logDriver: journald
  memFree: 36294004736
  memTotal: 67297546240
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.14.0-1.1
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.14.0
    package: netavark-1.14.1-1.1
    path: /usr/lib/podman/netavark
    version: netavark 1.14.1
  ociRuntime:
    name: runc
    package: runc-1.3.0-1.1
    path: /usr/bin/runc
    version: |-
      runc version 1.3.0
      commit: 1.3.0-1-0-gcdc48a6-dirty
      spec: 1.2.1
      go: go1.24.2
      libseccomp: 2.5.6
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-2025_05_12.8ec1341-1.1
    version: ""
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 67296555008
  swapTotal: 67296555008
  uptime: 18h 57m 57.00s (Approximately 0.75 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/kallisti5/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/kallisti5/.local/share/containers/storage
  graphRootAllocated: 935996204032
  graphRootUsed: 100507894272
  graphStatus:
    Backing Filesystem: <unknown>
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 14
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/kallisti5/.local/share/containers/storage/volumes
version:
  APIVersion: 5.4.2
  Built: 1745100254
  BuiltTime: Sat Apr 19 17:04:14 2025
  GitCommit: be85287fcf4590961614ee37be65eeb315e5d9ff
  GoVersion: go1.24.2
  Os: linux
  OsArch: linux/amd64
  Version: 5.4.2

Seen in dmesg:

68120.146153] overlayfs: upper fs does not support file handles, falling back to index=off.
[68120.146158] overlayfs: fs on '/home/kallisti5/.local/share/containers/storage/overlay/l/UNTZLTR62KFPVRV6UK36W3Y3Z4' does not support file handles, falling back to xino=off.

kallisti5 avatar May 21 '25 21:05 kallisti5

https://www.phoronix.com/news/OverlayFS-Case-Insensitive seems related. Nevermind. Likely seeing this one due to bcachefs. Mounted a btrfs filesystem at $HOME/.local/share/containers/storage which resolved it.

kallisti5 avatar May 22 '25 01:05 kallisti5