storage
storage copied to clipboard
containers
Rootless Podman with `zfs` Storage Driver -- "Error: cannot find root filesystem"
Issue Description
Rootless Podman is unable to list ZFS filesystems.
Here is the rootless storage.conf file:
[storage]
driver = "zfs"
[storage.options.zfs]
fsname = "zroot/containers"
mountopt = "nodev"
This is the error I get when attempting to do anything with rootless Podman:
$ podman system info
Error: cannot find root filesystem zroot/containers: exit status 1: "/usr/bin/zfs list -rHp -t filesystem -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset zroot/containers" => cannot open 'zroot/containers': dataset does not exist
When I run the command manually using the same user, I am able to list the filesystem just fine:
$ /usr/bin/zfs list -rHp -t filesystem -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset zroot/containers
zroot/containers - 290816 480957955968 /home/<user>/.local/share/containers lz4 filesystem - 0 290816 290816 137728 290816
~/.local/share/containers is owned by <user>:<user> and here are the ZFS permissions:
$ zfs allow zroot/containers
---- Permissions on zroot/containers ---------------------------------
Local+Descendent permissions:
user <user> create,destroy,mount,snapshot
Steps to reproduce the issue
$ podman system reset$ su -# zfs create -o mountpoint=/home/<user>/.local/share/containers -o dedup=on zroot/containers# zfs allow <user> create,destroy,mount,snapshot zroot/containers# chown <user>:<user> /home/<user>/.local/share/containersexit$ podman system info
Describe the results you received
After running podman system info (or any Podman command as a regular user):
$ podman system info
Error: cannot find root filesystem zroot/containers: exit status 1: "/usr/bin/zfs list -rHp -t filesystem -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset zroot/containers" => cannot open 'zroot/containers': dataset does not exist
Describe the results you expected
The zfs storage driver is able to "see" the dataset and returns the system info without error.
podman info output
# podman --version
podman version 4.7.1
# uname -a
Linux saturn 6.5.8-arch1-1 containers/podman#1 SMP PREEMPT_DYNAMIC Thu, 19 Oct 2023 22:52:14 +0000 x86_64 GNU/Linux
# cat /etc/os-release
NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/"
LOGO=archlinux-logo
Podman in a container
No
Privileged Or Rootless
Rootless
Upstream Latest Release
Yes
Additional environment details
No response
Additional information
Interestingly, using the zfs storage driver as root works fine on the same system:
# podman system info [63/4503]
host:
arch: amd64
buildahVersion: 1.32.0
cgroupControllers:
- cpuset
- cpu
- io
- memory
- hugetlb
- pids
- rdma
- misc
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: /usr/bin/conmon is owned by conmon 1:2.1.8-1
path: /usr/bin/conmon
version: 'conmon version 2.1.8, commit: 00e08f4a9ca5420de733bf542b930ad58e1a7e7d'
cpuUtilization:
idlePercent: 99.98
systemPercent: 0.01
userPercent: 0
cpus: 40
databaseBackend: boltdb
distribution:
distribution: arch
version: unknown
eventLogger: journald
freeLocks: 2048
hostname: saturn
idMappings:
gidmap: null
uidmap: null
kernel: 6.5.8-arch1-1
linkmode: dynamic
logDriver: journald
memFree: 132329963520
memTotal: 135066349568
networkBackend: netavark
networkBackendInfo:
backend: netavark
dns:
package: /usr/lib/podman/aardvark-dns is owned by aardvark-dns 1.8.0-1
path: /usr/lib/podman/aardvark-dns
version: aardvark-dns 1.8.0
package: /usr/lib/podman/netavark is owned by netavark 1.8.0-1
path: /usr/lib/podman/netavark
version: netavark 1.8.0
ociRuntime:
name: crun
package: /usr/bin/crun is owned by crun 1.10-1
path: /usr/bin/crun
version: |-
crun version 1.10
commit: c053c83c57551bca13ead8600237341818975974
rundir: /run/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
pasta:
executable: ""
package: ""
version: ""
remoteSocket:
exists: false
path: /run/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: false
seccompEnabled: true
seccompProfilePath: /etc/containers/seccomp.json
selinuxEnabled: false
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.2-1
version: |-
slirp4netns version 1.2.2
commit: 0ee2d87523e906518d34a6b423271e4826f71faf
libslirp: 4.7.0
SLIRP_CONFIG_VERSION_MAX: 4
libseccomp: 2.5.4
swapFree: 0
swapTotal: 0
uptime: 1h 57m 42.00s (Approximately 0.04 days)
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
- ipvlan
volume:
- local
registries: {}
store:
configFile: /etc/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: zfs
graphOptions: {}
graphRoot: /var/lib/containers/storage
graphRootAllocated: 480958218240
graphRootUsed: 262144
graphStatus:
Compression: lz4
Parent Dataset: zroot/var/lib/containers
Parent Quota: "no"
Space Available: "480957980544"
Space Used By Parent: "241664"
Zpool: zroot
Zpool Health: ONLINE
imageCopyTmpDir: /var/tmp
imageStore:
number: 0
runRoot: /run/containers/storage
transientStore: false
volumePath: /var/lib/containers/storage/volumes
version:
APIVersion: 4.7.1
Built: 1696583554
BuiltTime: Fri Oct 6 05:12:34 2023
GitCommit: ef83eeb9c7482826672f3efa12db3d61c88df6c4-dirty
GoVersion: go1.21.1
Os: linux
OsArch: linux/amd64
Version: 4.7.1
No upstream maintainers work on the zfs, so it is not likely that upstream can help you. Perhaps people in the community. We recommend everyone use overlayfs on top of whatever file system you have.
No upstream maintainers work on the zfs, so it is not likely that upstream can help you. Perhaps people in the community. We recommend everyone use overlayfs on top of whatever file system you have.
@rhatdan: Please clarify. Do you mean there are no podman devs who support zfs storage at the moment? Using overlayfs where zfs is native is not desirable.
I came to report the same bug but with a minor variation in the error:
$ STORAGE_DRIVER=zfs podman system info
Error: cannot find root filesystem rpool/var/lib/containers/storage-<user>: exit status 1: "/usr/sbin/zfs fs list -rHp -t filesystem -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset rpool/var/lib/containers/storage-<user>" => cannot open 'rpool/var/lib/containers/storage-<user>': dataset does not exist
If the error code is derived from the actual command executed, then it should be noted that: /usr/sbin/zfs fs list -rHp... is not a valid command.
I don't know where to look but I assume someone familiar with the code could verify it isn't a basic fat-finger error with minimal effort.
zfs is not native in the upstream kernel and none of the core development team use it.
I see. How did the current level of support develop? Is the current level of support going to be deprecated?
Since this affects the basic functionality and anything ZFS related is "can't fix/won't fix" some note should be made in the docs so that for those of us who consider this a critical function won't waste our time.
This is a community project, if community wants to support it, then they need to step up. The volunteers or people who are paid to work on this stuff do not work on zfs.
Understood and agreed. Unfortunately I posess neither the required skills or bandwidth to fix.
As it stands, rootless ZFS support is non-functional. This should be documented somewhere so that users are aware. The volunteers or people who are paid to work on this stuff do have access to update the docs. It isn't a big ask to make a footnote in the appropriate location. Maybe an additonal note that devs with ZFS experience would be especially welcome...
podman is running the wrong command:
$ podman system reset -f
Error: cannot find root filesystem rpool/containers_user: exit status 1: "/usr/sbin/zfs fs list -rHp -t filesystem -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset rpool/containers_user" => cannot open 'rpool/containers_user': dataset does not exist
The correct command is: zfs list -Hp -t filesystem -o name,origin,used,available,mountpoint,compression,type,volsize,quota,referenced,written,logicalused,usedbydataset rpool/containers_user
This was on Debian 12's 4.3.1 though.
Interested in opening a PR?
Figuring out first how Ubuntu 24.04 makes podman work with ZFS as backing storage with overlay as driver. The bug was in Debian 12's podman 4.3.1 though.
I believe this bug is no longer valid on latest podman. Podman even works on FreeBSD which also uses ZFS.
I believe this bug is no longer valid on latest podman. Podman even works on FreeBSD which also uses ZFS.
This is good news. Unfortunately all of our relavent systems are running Debian 12. Any idea in what version this was fixed?
I believe this bug is no longer valid on latest podman. Podman even works on FreeBSD which also uses ZFS.
This is good news. Unfortunately all of our relavent systems are running Debian 12. Any idea in what version this was fixed?
Maybe I'm wrong. Ubuntu 24.04 has podman 4.9.3 working with ZFS as backing filesystem, not as graph driver. I don't know if it uses fuse-overlayfs to get it working because trying the same without FUSE in Debian 12 fails.
FreeBSD does use podman 4.8.3 with ZFS as graph driver. Only runs as root, though.
I prefer to use VFS as graph driver over fuse-overlayfs with ZFS as backing filesystem. This is my /etc/containers/storage.conf:
[storage]
driver = "vfs"
graphroot = "/var/lib/containers/storage"
runroot = "/run/containers/storage"
You have to remove root and user storage directories and run podman system reset -f. You know the drill.
For reasons that are not entirely clear to me, the general consensus (between Docker and ZFS devs) is that the FUSE ZFS driver is neither performant nor entirely stable compared to the native ZOL driver. Also, VFS does not allow for the performance / storage gains of using copy-on-write, which is a core driver for using ZFS. The clone/snapshot/clone mechanism is really excellent for a PaaS environment.
What we are looking for is rootless, daemonless containers leveraging ZFS. Podman in theory does all this, but the rootless containers fail when asked to use the ZFS driver. If we run them as root they work great...but the inherent security issues remain. Since it looks like getting ZFS on rootless containers to work on Debian 12 is problematic, we are investigating moving over to Docker instead.
@ricardobranco777 Please do post back if you find a version of podman where rootless containers are working correctly with the native ZoL ZFS driver
Does Docker work with zfs when run in rootless mode? Or are you running with rootful Docker?
Does Docker work with zfs when run in rootless mode? Or are you running with rootful Docker?
Haven't tried rootless Docker with ZFS.
That is the issue, you drop Podman because zfs does not work with rootless mode, but then switch to Docker in rootful mode.
Does Docker work with zfs when run in rootless mode? Or are you running with rootful Docker?
It appears not, which then brings us right back to podman...so we will not be migrating and will have to weigh the benefits of having rootless containers vs the performance and scaling implications of using vfs or overlayfs riding on zfs.
I'll keep an eye on this issue in hopes someone with ZFS chops takes interest and makes our dreams come true.