Podman is not able to create containers with additional image store.

[mahendra77024]

Issue Description

Hi ,

I have been trying to set up shared location for storing container images. So that, all users on system can leverage these images to create containers rather than pulling container images again. I have followed below blog and able to list the images but while creating containers it's failed with below error.

$podman run registry.fedoraproject.org/fedora ls -latr / Error: creating container storage: creating read-write layer with ID "6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555": potentially insufficient UIDs or GIDs available in user namespace (requested 65534:65534 for /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": chown /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff: invalid argument

https://www.redhat.com/sysadmin/image-stores-podman

Steps to reproduce the issue

#podman --root /var/lib/containers/test-storage pull fedora #chmod -R a+rx /var/lib/containers/test-storage switch to rootless user

create a storage.conf in ~/.config/containers/storage.conf

[storage] driver = "overlay"

[storage.options] additionalimagestores = [ "/var/lib/containers/test-storage",]

[storage.options.overlay] mount_program = "/usr/bin/fuse-overlayfs"

$ podman images REPOSITORY TAG IMAGE ID CREATED SIZE R/O registry.fedoraproject.org/fedora latest ec42546bb614 3 hours ago 233 MB true registry.access.redhat.com/ubi9 latest 3b63310310b9 2 weeks ago 220 MB true

$podman run registry.fedoraproject.org/fedora ls -latr / Error: creating container storage: creating read-write layer with ID "6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555": potentially insufficient UIDs or GIDs available in user namespace (requested 65534:65534 for /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": chown /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff: invalid argument

Describe the results you received

$podman run registry.fedoraproject.org/fedora ls -latr / Error: creating container storage: creating read-write layer with ID "6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555": potentially insufficient UIDs or GIDs available in user namespace (requested 65534:65534 for /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": chown /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff: invalid argument

Describe the results you expected

when run podman run registry.fedoraproject.org/fedora ls -latr / , it should be able to create with container without any errors as image is available in shared location with readonly mode.

podman info output

$podman info
host:
  arch: amd64
  buildahVersion: 1.31.5
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.8-1.module+el8.9.0+21697+6a5e98e7.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: 57ed23ee47beaf9a72b28f5666fab095a9ad4a38'
  cpuUtilization:
    idlePercent: 91.79
    systemPercent: 2.31
    userPercent: 5.9
  cpus: 2
  databaseBackend: boltdb
  distribution:
    distribution: '"rhel"'
    version: "8.9"
  eventLogger: file
  freeLocks: 2045
  hostname: xyz.example.com
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 669
      size: 1
    uidmap:
    - container_id: 0
      host_id: 85617
      size: 1
  kernel: 4.18.0-513.24.1.el8_9.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 1786728448
  memTotal: 8059088896
  networkBackend: cni
  networkBackendInfo:
    backend: cni
    dns:
      package: podman-plugins-4.6.1-9.module+el8.9.0+21697+6a5e98e7.x86_64
      path: /usr/libexec/cni/dnsname
      version: |-
        CNI dnsname plugin
        version: 1.3.1
        commit: unknown
    package: containernetworking-plugins-1.3.0-8.module+el8.9.0+21697+6a5e98e7.x86_64
    path: /usr/libexec/cni
  ociRuntime:
    name: runc
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.1.12
      spec: 1.0.2-dev
      go: go1.20.12
      libseccomp: 2.5.2
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    path: /tmp/podman-run-85617/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.1-1.module+el8.9.0+21697+6a5e98e7.x86_64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 1945104384
  swapTotal: 2147479552
  uptime: 312h 5m 19.00s (Approximately 13.00 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/users/xx/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.imagestore: /var/lib/containers/test-storage
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.12-1.module+el8.9.0+21697+6a5e98e7.x86_64
      Version: |-
        fusermount3 version: 3.3.0
        fuse-overlayfs: version 1.12
        FUSE library version 3.3.0
        using FUSE kernel interface version 7.26
  graphRoot: /home/users/xx/.local/share/containers/storage
  graphRootAllocated: 17169383424
  graphRootUsed: 221696000
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /tmp/containers-user-85617/containers
  transientStore: false
  volumePath: /home/users/xx/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.1
  Built: 1713524958
  BuiltTime: Fri Apr 19 04:09:18 2024
  GitCommit: ""
  GoVersion: go1.20.12
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

$ podman images REPOSITORY TAG IMAGE ID CREATED SIZE R/O registry.fedoraproject.org/fedora latest ec42546bb614 3 hours ago 233 MB true registry.access.redhat.com/ubi9 latest 3b63310310b9 2 weeks ago 220 MB true

$ cat /etc/redhat-release Red Hat Enterprise Linux release 8.9 (Ootpa) $ podman version Client: Podman Engine Version: 4.6.1 API Version: 4.6.1 Go Version: go1.20.12 Built: Fri Apr 19 04:09:18 2024 OS/Arch: linux/amd64 $ cat ~/.config/containers/storage.conf [storage] driver = "overlay"

[storage.options] additionalimagestores = [ "/var/lib/containers/test-storage",]

[storage.options.overlay] mount_program = "/usr/bin/fuse-overlayfs"

$podman info host: arch: amd64 buildahVersion: 1.31.5 cgroupControllers: [] cgroupManager: cgroupfs cgroupVersion: v1 conmon: package: conmon-2.1.8-1.module+el8.9.0+21697+6a5e98e7.x86_64 path: /usr/bin/conmon version: 'conmon version 2.1.8, commit: 57ed23ee47beaf9a72b28f5666fab095a9ad4a38' cpuUtilization: idlePercent: 91.79 systemPercent: 2.31 userPercent: 5.9 cpus: 2 databaseBackend: boltdb distribution: distribution: '"rhel"' version: "8.9" eventLogger: file freeLocks: 2045 hostname: xyz.example.com idMappings: gidmap: - container_id: 0 host_id: 669 size: 1 uidmap: - container_id: 0 host_id: 85617 size: 1 kernel: 4.18.0-513.24.1.el8_9.x86_64 linkmode: dynamic logDriver: k8s-file memFree: 1786728448 memTotal: 8059088896 networkBackend: cni networkBackendInfo: backend: cni dns: package: podman-plugins-4.6.1-9.module+el8.9.0+21697+6a5e98e7.x86_64 path: /usr/libexec/cni/dnsname version: |- CNI dnsname plugin version: 1.3.1 commit: unknown package: containernetworking-plugins-1.3.0-8.module+el8.9.0+21697+6a5e98e7.x86_64 path: /usr/libexec/cni ociRuntime: name: runc package: Unknown path: /usr/bin/runc version: |- runc version 1.1.12 spec: 1.0.2-dev go: go1.20.12 libseccomp: 2.5.2 os: linux pasta: executable: "" package: "" version: "" remoteSocket: path: /tmp/podman-run-85617/podman/podman.sock security: apparmorEnabled: false capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: true seccompEnabled: true seccompProfilePath: /usr/share/containers/seccomp.json selinuxEnabled: false serviceIsRemote: false slirp4netns: executable: /usr/bin/slirp4netns package: slirp4netns-1.2.1-1.module+el8.9.0+21697+6a5e98e7.x86_64 version: |- slirp4netns version 1.2.1 commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194 libslirp: 4.4.0 SLIRP_CONFIG_VERSION_MAX: 3 libseccomp: 2.5.2 swapFree: 1945104384 swapTotal: 2147479552 uptime: 312h 5m 19.00s (Approximately 13.00 days) plugins: authorization: null log:

• k8s-file
• none
• passthrough
• journald network:
• bridge
• macvlan
• ipvlan volume:
• local registries: search:
• registry.access.redhat.com
• registry.redhat.io
• docker.io store: configFile: /home/users/xx/.config/containers/storage.conf containerStore: number: 0 paused: 0 running: 0 stopped: 0 graphDriverName: overlay graphOptions: overlay.imagestore: /var/lib/containers/test-storage overlay.mount_program: Executable: /usr/bin/fuse-overlayfs Package: fuse-overlayfs-1.12-1.module+el8.9.0+21697+6a5e98e7.x86_64 Version: |- fusermount3 version: 3.3.0 fuse-overlayfs: version 1.12 FUSE library version 3.3.0 using FUSE kernel interface version 7.26 graphRoot: /home/users/xx/.local/share/containers/storage graphRootAllocated: 17169383424 graphRootUsed: 221696000 graphStatus: Backing Filesystem: xfs Native Overlay Diff: "false" Supports d_type: "true" Using metacopy: "false" imageCopyTmpDir: /var/tmp imageStore: number: 2 runRoot: /tmp/containers-user-85617/containers transientStore: false volumePath: /home/users/xx/.local/share/containers/storage/volumes version: APIVersion: 4.6.1 Built: 1713524958 BuiltTime: Fri Apr 19 04:09:18 2024 GitCommit: "" GoVersion: go1.20.12 Os: linux OsArch: linux/amd64 Version: 4.6.1

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[giuseppe]

how have you pulled the images in the shared store? Have you used fuse-overlays and set force_mask = "shared"?

[mahendra77024]

Hi @giuseppe ,

I ran this command as root user "podman --root /var/lib/containers/test-storage pull fedora" to pull the container image. I did set force_mask = "shared" in storage.conf but it didn't work

[giuseppe]

have you configured fuse-overlayfs in the storage.conf file?

[mahendra77024]

Yes I did configure. Please refer postman info and steps to reproduce section for more information. Thanks

[mahendra77024]

@giuseppe any help on this would be appreciated

[snowhanse]

Seems related to discussion here: https://github.com/containers/podman/issues/19827 ?

[mahendra77024]

@giuseppe podman doesn't support shared image store for running containers with rootless user?

[giuseppe]

it does, as long as extended attributes are supported by the underlying file system

[giuseppe]

I ran this command as root user "podman --root /var/lib/containers/test-storage pull fedora" to pull the container image. I did set force_mask = "shared" in storage.conf but it didn't work

what storage.conf? /etc/containers/storage.conf?

[mahendra77024]

@giuseppe I'm running podman as rootless user. So, The storage.conf that it is referring is $HOME/.config/containers/storage.conf

Here is the steps to reproduce issue

#podman --root /var/lib/containers/test-storage pull fedora #chmod -R a+rx /var/lib/containers/test-storage

switch to rootless user

create a storage.conf in $HOME/.config/containers/storage.conf

[storage] driver = "overlay"

[storage.options] additionalimagestores = [ "/var/lib/containers/test-storage",]

[storage.options.overlay] mount_program = "/usr/bin/fuse-overlayfs"

$ podman images REPOSITORY TAG IMAGE ID CREATED SIZE R/O registry.fedoraproject.org/fedora latest ec42546bb614 3 hours ago 233 MB true registry.access.redhat.com/ubi9 latest 3b63310310b9 2 weeks ago 220 MB true

$podman run registry.fedoraproject.org/fedora ls -latr / Error: creating container storage: creating read-write layer with ID "6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555": potentially insufficient UIDs or GIDs available in user namespace (requested 65534:65534 for /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": chown /home/users/xx/.local/share/containers/storage/overlay/6832abfe808fde7e470689076555b5eef07d91c96335c1ded08f6b42d3690555/diff: invalid argument

[giuseppe]

how does the /etc/containers/storage.conf file look like when you pull the image as root?

You need to set force_mask = "shared" there

[mahendra77024]

@giuseppe , below is the storage.conf file for root. I do see i have already configured force_mask = "shared".

grep -v ^# /etc/containers/storage.conf

[storage]

driver = "overlay"

runroot = "/run/containers/storage"

graphroot = "/var/lib/containers/storage"

[storage.options] ignore_chown_errors = "true" additionalimagestores = ["/var/lib/containers/test-storage"]

pull_options = {enable_partial_images = "false", use_hard_links = "false", ostree_repos=""}

remap-uids = "0:1668442479:65536" remap-gids = "0:1668442479:65536"

[storage.options.overlay] ignore_chown_errors = ""

mount_program = "/usr/bin/fuse-overlayfs"

mountopt = "nodev,metacopy=on"

force_mask = "shared"

[storage.options.thinpool]

[giuseppe]

sorry, I've missed this part earlier:

idMappings:
gidmap:
- container_id: 0
host_id: 669
size: 1
uidmap:
- container_id: 0
host_id: 85617
size: 1

You've only one ID available in your user namespace.

You can try to workaround this limitation using the squash_to_root option for fuse-overlayfs. Try adding it to the mountopt field in the ~/.config/containers/storage.conf file.

[mahendra77024]

@giuseppe I'm not sure if i'm doing something wrong. I have tested with mountopt = "squash_to_root" but it didn't work. Could you please see if storage.conf looks fine?

$ cat ~/.config/containers/storage.conf [storage] driver = "overlay"

[storage.options] additionalimagestores = [ "/var/lib/containers/test-storage",]

[storage.options.overlay] mount_program = "/usr/bin/fuse-overlayfs" mountopt = "squash_to_root" $ podman images REPOSITORY TAG IMAGE ID CREATED SIZE R/O registry.fedoraproject.org/fedora latest bd78a74aa43c 6 hours ago 233 MB true

$ podman run bd78a74aa43c Error: creating container storage: creating read-write layer with ID "c39090d5c4185af60c2e94b0643a4ed3f74342b17e6617215d686bec062ecf41": potentially insufficient UIDs or GIDs available in user namespace (requested 65534:65534 for /home/users/xx/.local/share/containers/storage/overlay/c39090d5c4185af60c2e94b0643a4ed3f74342b17e6617215d686bec062ecf41/diff): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": chown /home/users/xx/.local/share/containers/storage/overlay/c39090d5c4185af60c2e94b0643a4ed3f74342b17e6617215d686bec062ecf41/diff: invalid argument

[giuseppe]

@giuseppe I'm not sure if i'm doing something wrong. I have tested with mountopt = "squash_to_root" but it didn't work. Could you please see if storage.conf looks fine?

thanks. The configuration looks correctly, but I don't remember ever testing the combination squash_to_root and force_mode, so it might just be broken

[mahendra77024]

Thanks @giuseppe. Any insights how this can be addressed.

[mahendra77024]

@giuseppe is there anything that can be done to resolve this issue ? Thanks

[giuseppe]

can you configure multiple additional IDs for your user?

Is your user listed under /etc/subuid and /etc/subgid?

[mahendra77024]

Hi @giuseppe , if i map additional ID's for my user that works but we can't use this as we use LDAP and have many other users logging in to the system. that's one of the reason why i'm trying to use shared storage since they fill up disk space.

[giuseppe]

another alternative you've is to rewrite the mode extended attribute and force every file to be owned by root, so its value should be 0:0:$MODE

[giuseppe]

or you can try to use force_mode also for the rootless user, so that the file ownership is forced to root

[mahendra77024]

@giuseppe Could you please let me know how to do this ? Is this something i need to make a change in storage.conf? Thanks

[giuseppe]

yes, you need to change it in the ~/.config/containers/storage.conf file

[mahendra77024]

Hi @giuseppe , Could you please share full configuration or example of "force_mode" that has to be set in ~/.config/containers/storage.conf? I couldn't find any document for force_mode in storage.conf. Thanks

[giuseppe]

you've already used force_mode in /etc/containers/storage.conf right? Otherwise the store could not be used by rootless. Now I am suggesting you use the same setting for ~/.config/containers/storage.conf.

[storage]
  driver = "overlay"

[storage.options.overlay]
  mount_program = "/usr/bin/fuse-overlayfs"
  force_mask = "shared"

[mahendra77024]

@giuseppe , I guess i have already tried this option earlier and also now but i don't see that's working.

$ podman images REPOSITORY TAG IMAGE ID CREATED SIZE R/O registry.fedoraproject.org/fedora latest 85370d566a4c 2 days ago 233 MB true bd78a74aa43c 8 days ago 233 MB true docker.io/library/alpine latest 1d34ffeaf190 4 weeks ago 8.08 MB true

$ cat ~/.config/containers/storage.conf [storage] driver = "overlay" [storage.options] ignore_chown_errors = "true" additionalimagestores = [ "/var/lib/containers/test-storage",] [storage.options.overlay] mount_program = "/usr/bin/fuse-overlayfs" force_mask = "shared"

$ podman run registry.fedoraproject.org/fedora:latest Error: creating container storage: creating read-write layer with ID "8b8e1b220a3fff71a898344390fda3efa8bc89b9ef9536f2809420aa44ab7d8e": potentially insufficient UIDs or GIDs available in user namespace (requested 65534:65534 for /home/users/xx/.local/share/containers/storage/overlay/8b8e1b220a3fff71a898344390fda3efa8bc89b9ef9536f2809420aa44ab7d8e/diff): Check /etc/subuid and /etc/subgid if configured locally and run "podman system migrate": chown /home/users/xx/.local/share/containers/storage/overlay/8b8e1b220a3fff71a898344390fda3efa8bc89b9ef9536f2809420aa44ab7d8e/diff: invalid argument

[mahendra77024]

@giuseppe , If you think this is a bug, can it be added to future request ? Or may be you can try to reproduce this issue from your end that may give you more insights

[mahendra77024]

Hey @giuseppe , Just following up if you have any suggestions

[mahendraredwarasila-snc]

Hey @giuseppe , Do you have anything to say this can be fixed ?

[giuseppe]

sorry but I am looking at other issues at the moment and I won't have time to look at this one any time soon. If you want to see it fixed, please open a PR. It might need changes in fuse-overlayfs too