podman icon indicating copy to clipboard operation
podman copied to clipboard

Published 20 hours ago • verified publisher icon containers

Access Denied when running 'podman images' command

Open ankurmalhotra07 opened this issue 9 months ago • 4 comments

Issue Description

Describe your issue Want to use additional image stores as explained in this guide However, running into permission access denied issues when trying to run podman images/pull commands.

Steps to reproduce the issue

Steps to reproduce the issue

  1. Provision EFS share in AWS
  2. Mount EFS share mount -t efs -o tls fs-123...:/ /var/lib/mycontainers
  3. Pull image using podman time podman --root /var/lib/mycontainers pull docker.io/amazoncorretto:latest
  4. Make EFS share read only
  5. Run podman images

Describe the results you received

Describe the results you received podman images `+ podman images

Error: open /var/lib/mycontainers/overlay-images/images.lock: permission denied

script returned exit code 125`

Describe the results you expected

Describe the results you expected Images are displayed without any errors

podman info output

+ podman images

Error: open /var/lib/mycontainers/overlay-images/images.lock: permission denied

script returned exit code 125

Podman in a container

Yes

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

No response

ankurmalhotra07 avatar May 01 '24 16:05 ankurmalhotra07

please strace the command and report the failing syscall. You can do it by running strace -o /tmp/podman.log -f -v -s 1000 -Z podman images and attach the /tmp/podman.log file you get.

giuseppe avatar May 02 '24 09:05 giuseppe

@giuseppe here you go- `+ cat /tmp/podman.log

97 arch_prctl(0x3001 /* ARCH_??? */, 0x7ffe289839d0) = -1 EINVAL (Invalid argument)

97 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)

97 statfs("/selinux", 0x7ffe28983990) = -1 ENOENT (No such file or directory)

97 access("/etc/selinux/config", F_OK) = -1 ENOENT (No such file or directory)

97 epoll_ctl(4, EPOLL_CTL_ADD, 3, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=1045954561, u64=9213806973933846529}}) = -1 EPERM (Operation not permitted)

97 seccomp(SECCOMP_SET_MODE_STRICT, 0x1, NULL) = -1 EINVAL (Invalid argument)

97 seccomp(SECCOMP_SET_MODE_FILTER, SECCOMP_FILTER_FLAG_TSYNC, NULL) = -1 EFAULT (Bad address)

97 seccomp(SECCOMP_SET_MODE_FILTER, SECCOMP_FILTER_FLAG_LOG, NULL) = -1 EFAULT (Bad address)

97 seccomp(SECCOMP_SET_MODE_FILTER, SECCOMP_FILTER_FLAG_SPEC_ALLOW, NULL) = -1 EFAULT (Bad address)

97 seccomp(SECCOMP_SET_MODE_FILTER, SECCOMP_FILTER_FLAG_NEW_LISTENER, NULL) = -1 EFAULT (Bad address)

97 seccomp(SECCOMP_SET_MODE_FILTER, SECCOMP_FILTER_FLAG_TSYNC_ESRCH, NULL) = -1 EFAULT (Bad address)

97 futex(0x55e0d35e06c0, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

97 futex(0x55e0d35e06c0, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

99 futex(0xc00007e948, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

101 futex(0xc000100148, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

97 futex(0x55e0d35e06c0, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

101 futex(0xc000100148, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

100 futex(0xc00007ed48, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

100 futex(0x55e0d35e24d8, FUTEX_WAIT_PRIVATE, 2, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

104 futex(0x55e0d35e24d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)

100 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

100 futex(0x55e0d35e24d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)

100 futex(0xc00007ed48, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

97 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

100 futex(0x55e0d35e25d8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=100000}) = -1 ETIMEDOUT (Connection timed out)

100 futex(0x55e0d35e25d8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=100000}) = -1 ETIMEDOUT (Connection timed out)

97 futex(0x55e0d35e11a8, FUTEX_WAIT_PRIVATE, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

97 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

97 futex(0x55e0d35e11a8, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

105 futex(0x55e0d35e25d8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=100000}) = -1 ETIMEDOUT (Connection timed out)

97 futex(0x55e0d35e11a8, FUTEX_WAIT_PRIVATE, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

97 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

97 futex(0x55e0d35e11a8, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

97 futex(0x55e0d35e24d8, FUTEX_WAIT_PRIVATE, 2, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

97 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

97 futex(0x55e0d35e24d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)

97 futex(0x55e0d35e24d8, FUTEX_WAIT_PRIVATE, 2, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

97 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

97 futex(0x55e0d35e24d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)

99 futex(0xc00007e948, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

105 futex(0x55e0d35e25d8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=100000}) = -1 EAGAIN (Resource temporarily unavailable)

97 statfs("/sys/fs/cgroup/unified", 0xc0006bfc98) = -1 ENOENT (No such file or directory)

97 epoll_ctl(4, EPOLL_CTL_ADD, 3, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=1045954565, u64=9213806973933846533}}) = -1 EPERM (Operation not permitted)

97 newfstatat(AT_FDCWD, "/var/../run/containers", 0xc00011f148, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)

97 newfstatat(AT_FDCWD, "/var/lib/containers/storage", 0xc00011f628, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)

97 epoll_ctl(4, EPOLL_CTL_ADD, 3, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=1045954566, u64=9213806973933846534}}) = -1 EPERM (Operation not permitted)

97 newfstatat(AT_FDCWD, "/etc/containers/containers.conf.d", 0xc000146378, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)

97 epoll_ctl(4, EPOLL_CTL_ADD, 3, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=1045954570, u64=9213806973933846538}}) = -1 EPERM (Operation not permitted)

97 epoll_ctl(4, EPOLL_CTL_ADD, 3, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=1045954571, u64=9213806973933846539}}) = -1 EPERM (Operation not permitted)

97 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

105 futex(0xc000518148, FUTEX_WAIT_PRIVATE, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

105 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

106 futex(0xc000600148, FUTEX_WAIT_PRIVATE, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

106 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

99 futex(0xc00007e948, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

97 futex(0x55e0d35e11a8, FUTEX_WAIT_PRIVATE, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

97 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

97 futex(0x55e0d35e11a8, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

97 epoll_ctl(4, EPOLL_CTL_ADD, 3, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=1045954572, u64=9213806973933846540}}) = -1 EPERM (Operation not permitted)

100 futex(0x55e0d35e25d8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=100000}) = -1 ETIMEDOUT (Connection timed out)

97 epoll_ctl(4, EPOLL_CTL_ADD, 3, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=1045954573, u64=9213806973933846541}}) = -1 EPERM (Operation not permitted)

97 newfstatat(AT_FDCWD, "/etc/containers/containers.conf.d", 0xc00011ed38, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)

97 epoll_ctl(4, EPOLL_CTL_ADD, 3, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=1045954575, u64=9213806973933846543}}) = -1 EPERM (Operation not permitted)

97 epoll_ctl(4, EPOLL_CTL_ADD, 3, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=1045954576, u64=9213806973933846544}}) = -1 EPERM (Operation not permitted)

97 futex(0x55e0d35e11a8, FUTEX_WAIT_PRIVATE, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

97 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

102 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

102 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

102 futex(0xc00007f148, FUTEX_WAIT_PRIVATE, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

102 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

102 futex(0xc00007f148, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

97 futex(0x55e0d35e24d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)

97 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

97 rt_sigreturn({mask=[]}) = -1 EAGAIN (Resource temporarily unavailable)

105 futex(0x55e0d35e25d8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=100000}) = -1 ETIMEDOUT (Connection timed out)

97 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

97 futex(0x55e0d35e24d8, FUTEX_WAIT_PRIVATE, 2, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

97 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

97 futex(0x55e0d35e24d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)

97 futex(0x55e0d35e24d8, FUTEX_WAIT_PRIVATE, 2, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

97 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

97 futex(0x55e0d35e24d8, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)

105 futex(0x55e0d35e25d8, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=100000}) = -1 EAGAIN (Resource temporarily unavailable)

105 futex(0xc000518148, FUTEX_WAIT_PRIVATE, 0, NULL) = -1 EAGAIN (Resource temporarily unavailable)

100 newfstatat(AT_FDCWD, "/usr/libexec/podman/conmon", 0xc0004f6038, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/usr/local/libexec/podman/conmon", 0xc0004f6108, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/usr/local/lib/podman/conmon", 0xc0004f61d8, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/libpod", 0xc0004f6378, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage", 0xc0004f6448, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/run/libpod", 0xc0004f65e8, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/libpod/bolt_state.db", 0xc0004f6788, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/db.sql", 0x7fde054389b0, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/db.sql-journal", 0x7fde054399b0, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/db.sql-wal", 0x7fde054399b0, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/db.sql-journal", 0x7fde0543a8a0, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/db.sql-wal", 0x7fde0543a8a0, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/db.sql-journal", 0x7fde0543a8c0, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/db.sql-wal", 0x7fde0543a8c0, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/db.sql-journal", 0x7fde0543a8a0, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/db.sql-wal", 0x7fde0543a8a0, 0) = -1 ENOENT (No such file or directory)

100 --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=97, si_uid=0} ---

100 newfstatat(AT_FDCWD, "/var/run/containers/storage", 0xc0004f6b98, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/run/containers", 0xc0004f6c68, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/overlay", 0xc0004f6ed8, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/overlay/l", 0xc0004f72e8, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/overlay/l", 0xc0004f77c8, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/run/containers/storage/overlay", 0xc0004f7968, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/run/containers/storage/overlay", 0xc0004f7d78, 0) = -1 ENOENT (No such file or directory)

100 epoll_ctl(4, EPOLL_CTL_ADD, 8, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=1045954577, u64=9213806973933846545}}) = -1 EPERM (Operation not permitted)

100 quotactl(QCMD(Q_XSETQLIM, PRJQUOTA), "/var/lib/containers/storage/overlay/backingFsBlockDev", 459145140, {d_version=1, d_flags=FS_PROJ_QUOTA, d_fieldmask=0, d_id=459145140, d_blk_hardlimit=0, d_blk_softlimit=0, d_ino_hardlimit=0, d_ino_softlimit=0, d_bcount=0, d_icount=0, d_itimer=0, d_btimer=0, d_iwarns=0, d_bwarns=0, d_rtb_hardlimit=0, d_rtb_softlimit=0, d_rtbcount=0, d_rtbtimer=0, d_rtbwarns=0}) = -1 ENOSYS (Function not implemented)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/overlay-images", 0xc000570518, 0) = -1 ENOENT (No such file or directory)

100 openat(AT_FDCWD, "/var/lib/containers/storage/overlay-images/images.json", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/lib/containers/storage/overlay-containers", 0xc000570788, 0) = -1 ENOENT (No such file or directory)

100 newfstatat(AT_FDCWD, "/var/run/containers/storage/overlay-containers", 0xc000570928, 0) = -1 ENOENT (No such file or directory)

100 openat(AT_FDCWD, "/var/lib/containers/storage/overlay-containers/containers.json", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

100 openat(AT_FDCWD, "/var/lib/containers/storage/overlay-containers/volatile-containers.json", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

100 openat(AT_FDCWD, "/var/lib/mycontainers/overlay-images/images.lock", O_RDONLY|O_CREAT|O_CLOEXEC, 0644) = -1 EACCES (Permission denied)

106 +++ exited with 125 +++

104 +++ exited with 125 +++

103 +++ exited with 125 +++

102 +++ exited with 125 +++

101 +++ exited with 125 +++

100 +++ exited with 125 +++

99 +++ exited with 125 +++

98 +++ exited with 125 +++

105 +++ exited with 125 +++

97 +++ exited with 125 +++`

ankurmalhotra07 avatar May 02 '24 13:05 ankurmalhotra07 

100 openat(AT_FDCWD, "/var/lib/mycontainers/overlay-images/images.lock", O_RDONLY|O_CREAT|O_CLOEXEC, 0644) = -1 EACCES (Permission denied)

This error depends on your file system, it looks like it doesn't honor CAP_DAC_OVERRIDE, and the look up inside /var/lib/mycontainers fails for that reason. It is not something we can address in Podman

giuseppe avatar May 03 '24 11:05 giuseppe

did you run chmod -R 755 /var/lib/mycontainers?

giuseppe avatar May 03 '24 12:05 giuseppe

A friendly reminder that this issue had no activity for 30 days.

github-actions[bot] avatar Jun 03 '24 00:06 github-actions[bot]