siderolabs
feat: added additional zfs services to support encrypted volumes
This should work fine but I am not sure if the dependencies work that way. Please review...
With this config zfs filesystems with encrypted volumes and local keys (for example stored in /var) will be auto mounted.
This is an example from my test system:
root@worker2:~# df
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 975037276 13135376 961901900 2% /
tmpfs 65536 0 65536 0% /dev
overlay 256 256 0 100% /host
rootfs 16242936 104588 16138348 1% /host/.extra
devtmpfs 16242936 0 16242936 0% /host/dev
tmpfs 16295300 0 16295300 0% /dev/shm
tmpfs 16295300 0 16295300 0% /host/proc/acpi
tmpfs 16295300 0 16295300 0% /host/proc/scsi
efivarfs 192 92 96 50% /host/sys/firmware/efi/efivars
tmpfs 16295300 2628 16292672 1% /host/run
overlay 975037276 13135376 961901900 2% /run/containerd/io.containerd.runtime.v2.task/system/kubelet/rootfs
shm 65536 0 65536 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/c31fad6a21cd91e62facb7b20a61a020ac83600aeb67a51259623e4c93c5157c/shm
shm 65536 0 65536 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/c6851ae888bc2afad1c246e089c3902cad9a9362a87169a84bb7b476774414c4/shm
shm 65536 0 65536 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/a382de4b321b6324fe8cba0052ec3a372e3209963568f3b6cf9e9866f4bea094/shm
shm 65536 0 65536 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/9e6166cc734d1cdecc0763eafe2ec6ed8abc435e18af581c387ad3c0bf28c824/shm
overlay 975037276 13135376 961901900 2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/c31fad6a21cd91e62facb7b20a61a020ac83600aeb67a51259623e4c93c5157c/rootfs
overlay 975037276 13135376 961901900 2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/c6851ae888bc2afad1c246e089c3902cad9a9362a87169a84bb7b476774414c4/rootfs
overlay 975037276 13135376 961901900 2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/6b249ed4e115676c6ac82de699264f07701de5d26398e78d1e4b2b7a184e099f/rootfs
overlay 975037276 13135376 961901900 2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/a3afd6d213e959957cf5edf2b9db2ae42ddf660121c32399bcbc5ff9bc8886db/rootfs
overlay 975037276 13135376 961901900 2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/9e6166cc734d1cdecc0763eafe2ec6ed8abc435e18af581c387ad3c0bf28c824/rootfs
overlay 975037276 13135376 961901900 2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/a382de4b321b6324fe8cba0052ec3a372e3209963568f3b6cf9e9866f4bea094/rootfs
overlay 975037276 13135376 961901900 2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/84ee344d0c6af8b4d9a4be0f0cecb3667229043c61e98003f8f2edf7bcb73271/rootfs
overlay 975037276 13135376 961901900 2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/dce879378a9651f24fd5e9c72ae50bed34946d67ed57f0226ec4e8f832275488/rootfs
tmpfs 16295300 240 16295060 1% /host/system
overlay 256 256 0 100% /host/system/libexec/apid/apid
/dev/mapper/nvme0n1p3-encrypted 80544 5140 75404 7% /host/system/state
tmpfs 65536 0 65536 0% /host/tmp
overlay 16295300 240 16295060 1% /host/usr/etc/udev
/dev/mapper/nvme0n1p4-encrypted 975037276 13135376 961901900 2% /host/var
tmpfs 32291592 4 32291588 1% /host/var/lib/kubelet/pods/ecd04ced-2d4e-4e1a-a2e0-64b300abc082/volumes/kubernetes.io~secret/memberlist
tmpfs 32291592 0 32291592 0% /host/var/lib/kubelet/pods/a0b74911-c60d-4d3a-8187-863e39930ea9/volumes/kubernetes.io~secret/longhorn-grpc-tls
tmpfs 32291592 12 32291580 1% /host/var/lib/kubelet/pods/0d0689d2-e263-42b1-82a9-22ad3ade5ccc/volumes/kubernetes.io~projected/kube-api-access-qmg8n
tmpfs 32291592 12 32291580 1% /host/var/lib/kubelet/pods/eb279ec9-bf74-4102-ab89-3fe44cac45e8/volumes/kubernetes.io~projected/kube-api-access-m2v2n
tmpfs 32291592 12 32291580 1% /host/var/lib/kubelet/pods/b21d24d3-f21b-4dfe-ae43-4108b9241327/volumes/kubernetes.io~projected/kube-api-access-kwtmz
tmpfs 32291592 12 32291580 1% /host/var/lib/kubelet/pods/a0b74911-c60d-4d3a-8187-863e39930ea9/volumes/kubernetes.io~projected/kube-api-access-h5nvx
tmpfs 32291592 12 32291580 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 32291592 12 32291580 1% /host/var/lib/kubelet/pods/784bf64e-0f81-4e15-ae98-d98f055a5748/volumes/kubernetes.io~projected/kube-api-access-lhwhw
tmpfs 32291592 12 32291580 1% /host/var/lib/kubelet/pods/ecd04ced-2d4e-4e1a-a2e0-64b300abc082/volumes/kubernetes.io~projected/kube-api-access-s6lfj
tmpfs 32291592 12 32291580 1% /host/var/lib/kubelet/pods/fbbe174c-8888-4302-a8be-3ffaa7f8fc9b/volumes/kubernetes.io~projected/kube-api-access-jpsv5
tmpfs 32291592 12 32291580 1% /host/var/lib/kubelet/pods/55c9398f-7fa1-4a01-99fa-7ad13a04e506/volumes/kubernetes.io~projected/kube-api-access-hflnz
tmpfs 32291592 12 32291580 1% /host/var/lib/kubelet/pods/11bc1b90-1509-4084-bb11-85acd8276f0f/volumes/kubernetes.io~projected/kube-api-access-8t4zh
overlay 975037276 13135376 961901900 2% /host/etc/cni
overlay 975037276 13135376 961901900 2% /host/etc/kubernetes
overlay 975037276 13135376 961901900 2% /host/usr/libexec/kubernetes
overlay 975037276 13135376 961901900 2% /host/opt
overlay 16295300 240 16295060 1% /host/usr/local/lib/containers/tgtd
overlay 16295300 240 16295060 1% /host/usr/local/lib/containers/zpool-importer
overlay 16295300 240 16295060 1% /host/usr/local/lib/containers/iscsid
root@worker2:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank 1.57T 9.21T 96K /var/hddstorage
tank/private 195G 9.21T 195G /var/hddstorage/private
tank/public 1.38T 9.21T 1.38T /var/hddstorage/public
root@worker2:~# zfs load-key -a
2 / 2 key(s) successfully loaded
root@worker2:~# zfs mount -a
root@worker2:~# df
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 975037276 13135432 961901844 2% /
tmpfs 65536 0 65536 0% /dev
overlay 256 256 0 100% /host
rootfs 16242936 104588 16138348 1% /host/.extra
devtmpfs 16242936 0 16242936 0% /host/dev
tmpfs 16295300 0 16295300 0% /dev/shm
tmpfs 16295300 0 16295300 0% /host/proc/acpi
tmpfs 16295300 0 16295300 0% /host/proc/scsi
efivarfs 192 92 96 50% /host/sys/firmware/efi/efivars
tmpfs 16295300 2628 16292672 1% /host/run
overlay 975037276 13135432 961901844 2% /run/containerd/io.containerd.runtime.v2.task/system/kubelet/rootfs
shm 65536 0 65536 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/c31fad6a21cd91e62facb7b20a61a020ac83600aeb67a51259623e4c93c5157c/shm
shm 65536 0 65536 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/c6851ae888bc2afad1c246e089c3902cad9a9362a87169a84bb7b476774414c4/shm
shm 65536 0 65536 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/a382de4b321b6324fe8cba0052ec3a372e3209963568f3b6cf9e9866f4bea094/shm
shm 65536 0 65536 0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/9e6166cc734d1cdecc0763eafe2ec6ed8abc435e18af581c387ad3c0bf28c824/shm
overlay 975037276 13135432 961901844 2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/c31fad6a21cd91e62facb7b20a61a020ac83600aeb67a51259623e4c93c5157c/rootfs
overlay 975037276 13135432 961901844 2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/c6851ae888bc2afad1c246e089c3902cad9a9362a87169a84bb7b476774414c4/rootfs
overlay 975037276 13135432 961901844 2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/6b249ed4e115676c6ac82de699264f07701de5d26398e78d1e4b2b7a184e099f/rootfs
overlay 975037276 13135432 961901844 2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/a3afd6d213e959957cf5edf2b9db2ae42ddf660121c32399bcbc5ff9bc8886db/rootfs
overlay 975037276 13135432 961901844 2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/9e6166cc734d1cdecc0763eafe2ec6ed8abc435e18af581c387ad3c0bf28c824/rootfs
overlay 975037276 13135432 961901844 2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/a382de4b321b6324fe8cba0052ec3a372e3209963568f3b6cf9e9866f4bea094/rootfs
overlay 975037276 13135432 961901844 2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/84ee344d0c6af8b4d9a4be0f0cecb3667229043c61e98003f8f2edf7bcb73271/rootfs
overlay 975037276 13135432 961901844 2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/dce879378a9651f24fd5e9c72ae50bed34946d67ed57f0226ec4e8f832275488/rootfs
tmpfs 16295300 240 16295060 1% /host/system
overlay 256 256 0 100% /host/system/libexec/apid/apid
/dev/mapper/nvme0n1p3-encrypted 80544 5140 75404 7% /host/system/state
tmpfs 65536 8 65528 1% /host/tmp
overlay 16295300 240 16295060 1% /host/usr/etc/udev
/dev/mapper/nvme0n1p4-encrypted 975037276 13135432 961901844 2% /host/var
tmpfs 32291592 4 32291588 1% /host/var/lib/kubelet/pods/ecd04ced-2d4e-4e1a-a2e0-64b300abc082/volumes/kubernetes.io~secret/memberlist
tmpfs 32291592 0 32291592 0% /host/var/lib/kubelet/pods/a0b74911-c60d-4d3a-8187-863e39930ea9/volumes/kubernetes.io~secret/longhorn-grpc-tls
tmpfs 32291592 12 32291580 1% /host/var/lib/kubelet/pods/0d0689d2-e263-42b1-82a9-22ad3ade5ccc/volumes/kubernetes.io~projected/kube-api-access-qmg8n
tmpfs 32291592 12 32291580 1% /host/var/lib/kubelet/pods/eb279ec9-bf74-4102-ab89-3fe44cac45e8/volumes/kubernetes.io~projected/kube-api-access-m2v2n
tmpfs 32291592 12 32291580 1% /host/var/lib/kubelet/pods/b21d24d3-f21b-4dfe-ae43-4108b9241327/volumes/kubernetes.io~projected/kube-api-access-kwtmz
tmpfs 32291592 12 32291580 1% /host/var/lib/kubelet/pods/a0b74911-c60d-4d3a-8187-863e39930ea9/volumes/kubernetes.io~projected/kube-api-access-h5nvx
tmpfs 32291592 12 32291580 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 32291592 12 32291580 1% /host/var/lib/kubelet/pods/784bf64e-0f81-4e15-ae98-d98f055a5748/volumes/kubernetes.io~projected/kube-api-access-lhwhw
tmpfs 32291592 12 32291580 1% /host/var/lib/kubelet/pods/ecd04ced-2d4e-4e1a-a2e0-64b300abc082/volumes/kubernetes.io~projected/kube-api-access-s6lfj
tmpfs 32291592 12 32291580 1% /host/var/lib/kubelet/pods/fbbe174c-8888-4302-a8be-3ffaa7f8fc9b/volumes/kubernetes.io~projected/kube-api-access-jpsv5
tmpfs 32291592 12 32291580 1% /host/var/lib/kubelet/pods/55c9398f-7fa1-4a01-99fa-7ad13a04e506/volumes/kubernetes.io~projected/kube-api-access-hflnz
tmpfs 32291592 12 32291580 1% /host/var/lib/kubelet/pods/11bc1b90-1509-4084-bb11-85acd8276f0f/volumes/kubernetes.io~projected/kube-api-access-8t4zh
overlay 975037276 13135432 961901844 2% /host/etc/cni
overlay 975037276 13135432 961901844 2% /host/etc/kubernetes
overlay 975037276 13135432 961901844 2% /host/usr/libexec/kubernetes
overlay 975037276 13135432 961901844 2% /host/opt
overlay 16295300 240 16295060 1% /host/usr/local/lib/containers/tgtd
overlay 16295300 240 16295060 1% /host/usr/local/lib/containers/zpool-importer
overlay 16295300 240 16295060 1% /host/usr/local/lib/containers/iscsid
tank 9889162240 128 9889162112 1% /host/var/hddstorage
tank/private 10093786496 204624384 9889162112 3% /host/var/hddstorage/private
tank/public 11371600512 1482438400 9889162112 14% /host/var/hddstorage/public
I would really love to land some support for mounting/volumes in Talos 1.7 to avoid such workaround if possible.
Okay if you bring native support for that in 1.7 I am totally fine… if not please consider merging this workaround.
@smira I played arround with zfs a bit. Zfs encryption in talos is tricky at the moment:
- if you mount an encrypted volume (zfs load-key -a; zfs mount -a) using a debug pod the files are not accessible in another pod
- having a mix of encrypted and unencrypted zfs datasets results in none of them being automounted anymore
- normally all datasets are automated if you use zfs but sometimes this does not happen, so you have to call zfs mount --a , this is also problematic because pods might write data to some local fs instead of the real zfs fs. This can result in data loos.
I think there should be some tests and documentation for zfs usage to make this usable without these issues,
Hi there!
I wonder if we could provide some help or human bandwidth to make progress and clear the path forward as much as possible on that matter ? We really miss having at least basic at-rest ZFSencryption working on Talos nodes 😢
@smira following on what you said in june :
I would really love to land some support for mounting/volumes in Talos 1.7 to avoid such workaround if possible.
So I suppose in a post Talos 1.9 world, this means extending/enriching [VolumeConfig] (https://www.talos.dev/v1.9/reference/configuration/block/volumeconfig) but I guess this is no small task.. 😅
Maybe the solution proposed in this PR would not be such a bad temporary workaround until you got this figured out in a generic way in the volume management system ?
Or even a simpler variation of it, only issuing a zfs load-key -a immediately after zpool import and leaving everything else to the CSI (like openebs-zfs-localpv) that should be able to transparently create/mount zfs volumes within the pool without even having to know if the root pool is encrypted or not (since volumes created in an encrypted root pool inherit encryption config).
Also, I think this "only deal with the root pool" strategy should make all the issue raised by @runningman84 in https://github.com/siderolabs/extensions/pull/400#issuecomment-2158522646 void.
This key-loading step could be conditioned to the detection of encrypted zpool (easy to do with zfs and zpool binary).
Or even simply with a zfs load-key -n (performing a dry-run), see doc
Maybe not be such a bad temporary workaround until you got this figured in a generic way in the volume management system ?
We could work on a new PR and stress-test this a bit if that make sense ?
I contributed a service for the zfs extension (https://github.com/siderolabs/extensions/pull/513) which is included in Talos 1.9. This service runs zpool import -fal to import pools. So if you set keylocation on a pool to a file path stored on a Talos volume, you should get automatic encrypted pool import at boot. I store my keys in /var/zfs and use Talos disk encryption to secure that.
We don't have any bandwidth at the moment to work on ZFS yet, the Volume Management work is still ongoing.
This PR is stale because it has been open 45 days with no activity.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}