Error: creating /etc/mtab symlink: invalid argument

[axet]

Issue Description

podman failed to initialize rootless storage.

Steps to reproduce the issue

Steps to reproduce the issue:

1. create storage.conf

[storage]
driver = "overlay"
runroot = "$PWD/podman"
graphroot = "$PWD/podman"
rootless_storage_path = "$PWD/podman"

[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"

2. export CONTAINERS_STORAGE_CONF="$PWD/storage.conf"
3. podman run --rm registry.gitlab.steamos.cloud/proton/sniper/sdk:0.20240307.80401-0

Describe the results you received

Error: creating /etc/mtab symlink: invalid argument

Describe the results you expected

storage successfully initialized

podman info output

host:
  arch: amd64
  buildahVersion: 1.37.5
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.12-3_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: unknown'
  cpuUtilization:
    idlePercent: 88.54
    systemPercent: 3.79
    userPercent: 7.67
  cpus: 8
  databaseBackend: sqlite
  distribution:
    codename: trixie
    distribution: debian
    version: unknown
  eventLogger: journald
  freeLocks: 2048
  hostname: axet-desktop
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.11.7-amd64
  linkmode: dynamic
  logDriver: journald
  memFree: 3419398144
  memTotal: 33478434816
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.12.2-1_amd64
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.12.2
    package: netavark_1.12.1-3_amd64
    path: /usr/lib/podman/netavark
    version: netavark 1.12.1
  ociRuntime:
    name: crun
    package: crun_1.18.2-1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.18.2
      commit: 00ab38af875ddd0d1a8226addda52e1de18339b5
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt_0.0~git20241030.ee7d0b6-1_amd64
    version: |
      pasta 0.0~git20241030.ee7d0b6-1
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.1-1+b1_amd64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.8.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 19871469568
  swapTotal: 25769795584
  uptime: 20h 20m 33.00s (Approximately 0.83 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /media/axet/2TB/Games/proton903/podman/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs_1.13-1_amd64
      Version: |-
        fusermount3 version: 3.14.0
        fuse-overlayfs: version 1.13-dev
        FUSE library version 3.14.0
        using FUSE kernel interface version 7.31
  graphRoot: /media/axet/2TB/Games/proton903/podman
  graphRootAllocated: 2000381018112
  graphRootUsed: 1292271300608
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /media/axet/2TB/Games/proton903/podman/volumes
version:
  APIVersion: 5.2.5
  Built: 1729890366
  BuiltTime: Sat Oct 26 00:06:06 2024
  GitCommit: ""
  GoVersion: go1.23.2
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.5

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

Debian Trixie

Additional information

Happens everytime.

[giuseppe]

what is the underlying filesystem on /media/axet/2TB/Games/proton903?

[axet]

btrfs

[giuseppe]

thanks. I am still not able to reproduce locally.

Can you please share the output of cat /proc/self/mountinfo, then run podman unshare strace -e symlinkat -s 1000 -o log -f podman run --rm registry.gitlab.steamos.cloud/proton/sniper/sdk:0.20240307.80401-0 then attach the log file?

[axet]

log.gz

https://paste.debian.net/1337078/

[giuseppe]

thanks, still not able to reproduce here, even on Debian.

Was the store created by the same rootless user?

[axet]

Oh yes. Clean run. Acutally I'm using my own 'proton' script to fetch and build proton.

• https://gitlab.com/axet/homebin/-/blob/debian/proton

proton clone90exp
cd proton90exp
proton podman build

[giuseppe]

wonder if it is caused by fuse-overlayfs.

Any reason for using it instead of native overlay?

[axet]

root permission ofc. fuse-overlayfs for compiling proton root-less

[giuseppe]

you can use native overlay with rootless too

[axet]

With disabled:

#[storage.options.overlay]
#mount_program = "/usr/bin/fuse-overlayfs"

I got same:

48453 symlinkat("/proc/mounts", 9, "mtab" <unfinished ...>
48450 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=48447, si_uid=0} ---
48453 <... symlinkat resumed>)          = -1 EINVAL (Invalid argument)

[giuseppe]

you need to reset the store. Once fuse-overlayfs is used, it cannot be disabled because some metadata is stored differently

[axet]

Indeed it works with native overlayfs. Still a mystery to me how does overlayfs does not require root permission, but ok.

I'll keep it issue open, right?

[giuseppe]

native overlay works from a user namespace since linux v5.10.

I'll move it to fuse-overlayfs

[axet]

Build process of podman root-less environment failed because of lack of permissions. For example the following command failed:

cp -af /media/axet/2TB/local/proton903/openvr/bin/win32/openvr_api.dll /media/axet/2TB/local/proton903/build/build--local/dist/files/lib/wine/dxvk/openvr_api_dxvk.dll
/usr/bin/cp: preserving permissions for ‘/media/axet/2TB/local/proton903/build/build--local/dist/files/lib/wine/dxvk/openvr_api_dxvk.dll’: Invalid argument

But If I run same command without podman (or unshare -rm) command succeed.

It could be same issue affecting creating symlink under fuse-overlayfs. I guess podman unshare lack of permissions.

[giuseppe]

could you please run strace on the fuse-overlayfs to see what syscall is failing?

Run strace on fuse-overlayfs, run the failing command, then stop strace. Something like:

strace -o log -s 1000 -f -p $FUSE_OVERLAYFS_PID and please attach the resulting log file.

[axet]

Not sure what PID you are asking for. I ran ps aux|grep fuse during podman run --rm but I see no fuse-overlayfs processes.

[axet]

I even ran this: while true; do pidof fuse-overlayfs; done from root user during podman run --rm but got no pids.

And "Yes" I'm running updated storage with 'mount_program' set, so I'm getting "Error: creating /etc/mtab symlink: invalid argument".

I guess here is no fuse-overlayfs process started yet.

[giuseppe]

that is strange.

So are you getting the error from cp -af without any underlying fuse-overlayfs mount?

Can you run cp -af /media/axet/2TB/local/proton903/openvr/bin/win32/openvr_api.dll /media/axet/2TB/local/proton903/build/build--local/dist/files/lib/wine/dxvk/openvr_api_dxvk.dll under strace?

[axet]

podman unshare cp -a toolmanifest_runtime.vdf /media/axet/2TB/local/proton903-1/build/build--local/dist/toolmanifest.vdf
/usr/bin/cp: preserving permissions for ‘/media/axet/2TB/local/proton903-1/build/build--local/dist/toolmanifest.vdf’: Invalid argument

https://pastebin.com/q4BAPnX3

[giuseppe]

I am confused now, is there any fuse-overlayfs process running when you run the cp command?

To catch the cp failure, you need to run:

$ podman unshare strace -f -o log -s 1000 -v cp -a toolmanifest_runtime.vdf /media/axet/2TB/local/proton903-1/build/build--local/dist/toolmanifest.vdf

Is that podman command running on the host or are you in a nested container?

[axet]

Here are two prefixes 1) fuse 2) native. First time you told me to try fuse, last time you ask me to use native ("without any underlying fuse-overlayfs"). Right? So I did.

No podman running during test cp runs.

log.gz

[giuseppe]

if you running podman unshare strace -f -o log -s 1000 -v cp -a toolmanifest_runtime.vdf /media/axet/2TB/local/proton903-1/build/build--local/dist/toolmanifest.vdf on the host, then there is nothing that podman/fuse-overlayfs are doing except setting up the user namespace.

You can try with unshare -r --map-auto strace -f -o log -s 1000 -v cp -a toolmanifest_runtime.vdf /media/axet/2TB/local/proton903-1/build/build--local/dist/toolmanifest.vdf and it should have the same effect

[axet]

It does. Same error.

log.gz

[giuseppe]

yeah it looks like a similar error:

21305 fsetxattr(4, "system.posix_acl_access", "\2\0\0\0\1\0\6\0\377\377\377\377\4\0\5\0\377\377\377\377\10\0\7\0\377\377\377\377\20\0\6\0\377\377\377\377 \0\6\0\377\377\377\377", 44, 0) = -1 EINVAL (Invalid argument)

[axet]

# getfacl /media/axet/2TB/
getfacl: Removing leading '/' from absolute path names
# file: media/axet/2TB/
# owner: nobody
# group: nogroup
# flags: -s-
user::rwx
group::r-x
group:nogroup:rwx
mask::rwx
other::rwx
default:user::rwx
default:group::r-x
default:group:nogroup:rwx
default:mask::rwx
default:other::rwx

[axet]

Steps to reproduce:

• zramctl -f -s 100G
• mkfs.btrfs /dev/zram1
• mount /dev/zram1 /mnt/
• chattr +c -R /mnt
• chown -R nobody:nogroup "/mnt"
• chmod -R g+s "/mnt"
• setfacl -Rm d:g:nogroup:rwx,g:nogroup:rwx,d:o::rwx,o::rwx "/mnt"

Use 'proton' script to clone, configure and build.

[hmenke]

I'm getting a similar error. My Podman container fails to start with

Error: create /etc: no data available

and in strace I see

lgetxattr("/proc/self/fd/4/etc", "user.overlay.opaque", 0x7ffecbf0bc80, 16) = -1 ENODATA (No data available)

Backing filesystem of graphroot is GPFS, runroot is on tmpfs.