zfs
zfs copied to clipboard
openzfs
Unable to access snapshot files using /.zfs/snapshot/testsnapshot/
System information
| Type | Version/Name |
|---|---|
| Distribution Name | Fedora |
| Distribution Version | 30 |
| Linux Kernel | 5.2.18 |
| Architecture | x86_64 |
| ZFS Version | 0.8.2-1 |
| SPL Version | 0.8.2-1 |
Describe the problem you're observing
Unable to access snapshot files using /.zfs/snapshot/test/ Verified on freebsd 11, this is working. Also ubuntu module 0.7.5 this is working. I cant stress enough how serious this issue is.
Describe how to reproduce the problem
sudo bash echo "should be working" > /opt/testsnapshot zfs -r snapshot roottank@testsnapshot rm /opt/testsnapshot cd /.zfs/snapshot/testsnapshot/opt bash: cd: opt: Object is remote
Include any warning/errors/backtraces from the system logs
There is no visible errors. -->
I had the same problem with zfs 0.8.2, so I downgraded zfs to version 0.8.1 which allows accessing the root snapshots fine. BTW, snapshots of the other filesystems mounted under / such as /home/.zfs/snapshot/* can still be accessed with zfs 0.8.2.
The same problem here. Debian testing, zfs-dkms 0.8.2-2. Some additional info:
x220i:~% ls /.zfs/snapshot
2019-10-16-2057/ zfs-auto-snap_daily4-2019-10-13-0412/ zfs-auto-snap_frequent-2019-10-17-1930/ zfs-auto-snap_hourly3-2019-10-17-1517/
zfs-auto-snap_daily-2019-10-12-0306/ zfs-auto-snap_daily8-2019-10-01-0311/ zfs-auto-snap_frequent-2019-10-17-1945/ zfs-auto-snap_hourly6-2019-10-17-0617/
zfs-auto-snap_daily-2019-10-14-0311/ zfs-auto-snap_daily8-2019-10-09-0257/ zfs-auto-snap_hourly-2019-10-17-1617/ zfs-auto-snap_hourly6-2019-10-17-1217/
zfs-auto-snap_daily-2019-10-15-0320/ zfs-auto-snap_daily8-2019-10-17-0318/ zfs-auto-snap_hourly-2019-10-17-1717/ zfs-auto-snap_hourly6-2019-10-17-1817/
zfs-auto-snap_daily-2019-10-16-0316/ zfs-auto-snap_frequent-2019-10-17-1900/ zfs-auto-snap_hourly-2019-10-17-1917/
zfs-auto-snap_daily4-2019-10-05-0258/ zfs-auto-snap_frequent-2019-10-17-1915/ zfs-auto-snap_hourly3-2019-10-17-0917/
x220i:~% ls /.zfs/snapshot/zfs-auto-snap_daily8-2019-10-09-0257/
x220i:~% ls /.zfs/snapshot/zfs-auto-snap_daily8-2019-10-09-0257/.
ls: cannot access '/.zfs/snapshot/zfs-auto-snap_daily8-2019-10-09-0257/.': Object is remote
zsh: exit 2 ls
x220i:~% zfs list -o space -rt all
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
x220 159G 287G 0B 96K 0B 287G
x220/ROOT 159G 270G 0B 96K 0B 270G
x220/ROOT/debian 159G 270G 25.1G 245G 0B 0B
x220/ROOT/debian@zfs-auto-snap_daily8-2019-10-01-0311 - 1.47G - - - -
x220/ROOT/debian@zfs-auto-snap_daily4-2019-10-05-0258 - 1.21G - - - -
x220/ROOT/debian@zfs-auto-snap_daily8-2019-10-09-0257 - 2.15G - - - -
x220/ROOT/debian@zfs-auto-snap_daily-2019-10-12-0306 - 1.36G - - - -
x220/ROOT/debian@zfs-auto-snap_daily4-2019-10-13-0412 - 1009M - - - -
x220/ROOT/debian@zfs-auto-snap_daily-2019-10-14-0311 - 1.05G - - - -
x220/ROOT/debian@zfs-auto-snap_daily-2019-10-15-0320 - 2.01G - - - -
x220/ROOT/debian@zfs-auto-snap_daily-2019-10-16-0316 - 1.74G - - - -
x220/ROOT/debian@2019-10-16-2057 - 183M - - - -
x220/ROOT/debian@zfs-auto-snap_daily8-2019-10-17-0318 - 52.3M - - - -
x220/ROOT/debian@zfs-auto-snap_hourly6-2019-10-17-0617 - 42.8M - - - -
x220/ROOT/debian@zfs-auto-snap_hourly3-2019-10-17-0917 - 10.5M - - - -
x220/ROOT/debian@zfs-auto-snap_hourly6-2019-10-17-1217 - 17.3M - - - -
x220/ROOT/debian@zfs-auto-snap_hourly3-2019-10-17-1517 - 12.1M - - - -
x220/ROOT/debian@zfs-auto-snap_hourly-2019-10-17-1617 - 5.17M - - - -
x220/ROOT/debian@zfs-auto-snap_hourly-2019-10-17-1717 - 5.86M - - - -
x220/ROOT/debian@zfs-auto-snap_hourly6-2019-10-17-1817 - 37.6M - - - -
x220/ROOT/debian@zfs-auto-snap_frequent-2019-10-17-1915 - 5.95M - - - -
x220/ROOT/debian@zfs-auto-snap_hourly-2019-10-17-1917 - 6.00M - - - -
x220/ROOT/debian@zfs-auto-snap_frequent-2019-10-17-1930 - 11.8M - - - -
x220/ROOT/debian@zfs-auto-snap_frequent-2019-10-17-1945 - 13.8M - - - -
x220/ROOT/debian@zfs-auto-snap_frequent-2019-10-17-2000 - 6.68M - - - -
x220/home 159G 80.1M 0B 96K 0B 80.0M
x220/home/ark 159G 80.0M 0B 96K 0B 79.9M
x220/home/ark/tmp 159G 79.9M 0B 96K 0B 79.9M
x220/home/ark/tmp/android 159G 79.9M 0B 96K 0B 79.8M
x220/home/ark/tmp/android/music 159G 79.8M 0B 79.8M 0B 0B
x220/swap 162G 4.25G 0B 642M 3.63G 0B
x220/var 159G 12.4G 0B 96K 0B 12.4G
x220/var/tmp 159G 12.4G 0B 96K 0B 12.4G
x220/var/tmp/gobuild 159G 12.4G 0B 12.4G 0B 0B
I switced to 0.8.8-2 (from 0.7.12-2+deb10u1) at 2019-10-12. The snapshots were created automatically, before switch and after. All snapshots have the sensible size, I think, they're fully functional, except that I cannot access them.
PR #9384 which resolves #9381, still does not allow me to access snapshots under /.zfs/snapshot/. The reason might be that I am using initramfs which first mounts the root filesystem under /sysroot/ before doing switch_root. See my comments in #9381.
Same here. Arch Linux, kernel 5.3.7, with package zfs-dkms 0.8.2-1 (archzfs-dkms)
Snapdirs of non-root-filesystems can be accessed without problems, e.g.
ls /home/.zfs/snapshot/20191030-2155
works. journalctl -b -g zfs then shows lines such as:
Oct 31 10:37:04 Celsius systemd[6199]: home-.zfs-snapshot-20191030\x2d2155.mount: Succeeded.
Oct 31 10:37:04 Celsius systemd[1]: home-.zfs-snapshot-20191030\x2d2155.mount: Succeeded.
And grep -F .zfs /proc/mounts shows the mount for a while.
However, trying ls /.zfs/snapshot/20191030-2155 yields no output and no error message.
Doing ls /.zfs/snapshot/20191030-2155/. (note the appended /.) yields the error:
ls: cannot access '/.zfs/snapshot/20191030-2155/.': Object is remote
And there are no associated entries in the output of journalctl -b -g zfs nor grep -F .zfs /proc/mounts.
Any hints on how to dig further?
I also get this error for any datasets that have mountpoint=legacy with 0.8.2.
I take that back. It's all my datasets that are mounted during initramfs that have the problem; my legacy mount that's mounted in stage 2 boot doesn't have this problem. Also, of course, if I create a new legacy mount, I can mount it and access its .zfs/snapshot/foo contents.
I have the same problem, it is a regression in 0.8.2 release (it worked fine in 0.8.1)
@Bronek you're likely seeing the issue resolved by PR #9384. We'll get the fix applied to 0.8.3.
Posting this in here since #9384 was closed:
Looks like I found a different kind of bug related to this. I've applied the patch listed above. (#9353 My rpool looks like this: rpool 1.27T 2.95T 176K none rpool/ROOT 145G 2.95T 50.1G / rpool/mysql 53.0G 2.95T 23.0G /var/lib/mysql rpool/mysql-log 5.79G 2.95T 1.44G /var/lib/mysql-log rpool/plexmediaserver 206G 2.95T 171G /var/lib/plexmediaserver rpool/spool 115G 2.95T 35.5G /var/spool rpool/virtual_machines 778G 2.95T 743G /var/lib/libvirt
I can see inside snapshots for rpool/ROOT: windwalker:~# ls /.zfs/snapshot/zfs-auto-snap_daily-2019-12-07-1107/ bin boot.tar.xz dead.letter etc lib lost+found mnt opt proc run share sys tmp var boot cdrom dev home lib64 media netshares path root sbin srv tftpboot usr
Looking inside snapshots for any other dataset in rpool results in "Too many levels of symbolic links" error: ls /var/lib/libvirt/.zfs/snapshot/zfs-auto-snap_daily-2019-12-07-1107/ ls: cannot access '/var/lib/libvirt/.zfs/snapshot/zfs-auto-snap_daily-2019-12-07-1107/': Too many levels of symbolic links
I just upgraded to 0.8.3, and the issue remains as described in my earlier post.
$ ls /home/.zfs/snapshot/
20200121-1930 20200123-0040 20200124-2134 20200126-0146
20200122-1325 20200123-1559 20200125-1946 20200126-2030
$ ls /home/.zfs/snapshot/20200126-2030 # works
ccorn
$ ls /.zfs/snapshot/
20200121-1930 20200123-0040 20200124-2134 20200126-0146
20200122-1325 20200123-1559 20200125-1946 20200126-2030
$ ls /.zfs/snapshot/20200126-2030 # no output
$ ls /.zfs/snapshot/20200126-2030/.
ls: cannot access '/.zfs/snapshot/20200126-2030/.': Object is remote
I too am having this issue on 0.8.3 (though in my case it affects multiple filesystems under my zroot pool, which contains /). This issue doesn't occur in any of my other pools that do not contain /.
$ ls /home/.zfs/snapshot/
2020-0122-0020_weekly 2020-0128-0100_hourly 2020-0128-1400_hourly
2020-0109-0000_daily 2020-0122-1839_before-pacman 2020-0128-0200_hourly 2020-0128-1400_quarterhour
$ ls /home/.zfs/snapshot/2020-0128-1400_quarterhour/ #returns nothing
$ ls /.zfs/snapshot/
2020-0108-0834_weekly 2020-0122-0020_weekly 2020-0128-0100_hourly 2020-0128-1400_hourly
2020-0109-0000_daily 2020-0122-1839_before-pacman 2020-0128-0200_hourly 2020-0128-1400_quarterhour
$ ls /.zfs/snapshot/2020-0128-1400_quarterhour/ #returns nothing
(I've abbreviated the output of ls when listing the snapshots available)
Would just like to chime in... seeing the same behavior on 0.8.3 and 0.8.2 zfs on Archlinux
One example from the logs:
1583461577 zfs_ctldir.c:1086:zfsctl_snapshot_mount(): Unable to automount /new_root/var/lib/docker/.zfs/snapshot/autosnap_2020-02-29_00:00:08_daily error=512
I can also confirm that by creating a non-root pool, I can access everything fine in the snapshot folder, so seems to be similar to ElvishJerrico. Any root derived filesystem though does not work as expected
I suspect this error occurs on systems with initramfs which chroot(2)s to new root filesystem and starts init process by running switch_root(8).
#10128 indicates that reverting 5a1bf9e 093bb64 solves this particular issue. Indeed that works for me. But that reopens the issues those commits were meant to solve. What now?
#10128 indicates that reverting 5a1bf9e 093bb64 solves this particular issue. Indeed that works for me. But that reopens the issues those commits were meant to solve. What now?
This was a good workaround for me.. yeah, as to what now, I'll leave that to the gurus :)
still happening on
zfs-0.8.4-r1-gentoo
zfs-kmod-0.8.4-r0-gentoo
non-root snapdirs are fine. but snapshots for root dataset shows up empty.
Filename: /lib/modules/5.5.7-200.fc31.x86_64/extra/zfs.ko.xz version: 0.8.4-1
Slightly different behaviour: -non root filesystems are okm (snapshot dirs visible) -root filesystem is ok immediately after boot (napshot dirs visible) but problems occur aftersome heavy work (compiling kernel)
found 2 workarounds (still on 0.8.4)
-
I've explicitly added my root dataset to fstab like this
zroot/ROOT/default / zfs defaults,noatime 0 0systemd (using generators) remounts / and/.zfs/snapshot/*content becomes accessible. -
another workaround, couple commands:
mkdir /sysroot
mount -o bind / /sysroot
after that snapshots will be accessible in /sysroot/.zfs/ just fine, but remain inaccessible via /.zfs/
/sysroot is the initial location dracut mounts root dataset to via zfsutil so somehow bind-mounting to this exact path or re-mounting to / after pivot_root restores expected behaviour.
Also experiencing this on Arch Linux, I can't access the root snapdirs. The workaround envolving mount -o bind to the original root mountpoint before chroot in initramfs works for me (its /new_root on my system as you can see below).
$ ls /.zfs/snapshot/a
$ cat /proc/spl/kstat/zfs/dbgmsg
...
1621944136 zfs_ctldir.c:1105:zfsctl_snapshot_mount(): Unable to automount /new_root/.zfs/snapshot/a error=512
openzfs 2.0.4, kernel 5.10.38
Same here on Ubuntu 20.04. zfs-0.8.3-1ubuntu12.9 zfs-kmod-0.8.3-1ubuntu12.7
Same here.
Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal 0.8.3-1ubuntu12.12 srcversion: 121E11703180683A8728698 name: zfs vermagic: 5.4.0-89-generic SMP mod_unload modversions
@nicman23 Not sure what legacy mounts are, but eventually. Im using:
zfs mount pool/vol
before I have set zfs set mountpoint=/dir pool/vol
Same issue on Ubuntu 21.10 running ZoL 2.0.6.
Both of @gyakovlev's workarounds worked for me; thank you so much for these!
Note that I could access .zfs/snapshot dirs for every filesystems but root (/.zfs/snapshot). I had the Unable to automount <snapdir> error=512 entries in /proc/spl/kstat/zfs/dbgmsg; my system is booted through ZFSbootmenu and dracut, which I strongly suspect uses a chroot mechanism similar to that described by @jhyeon above.
I am still experiencing the problem with the root dir from initramfs. 1666904221 zfs_ctldir.c:1106:zfsctl_snapshot_mount(): Unable to automount /new_root/.zfs/snapshot/autosnap_2022-10-27_20:00:04_hourly error=512
Bind mounting to /new_root allows access.
Confirmed this is still a bug on 2.1.7. Sending a snapshot to another dataset allows me to access the files but I cannot access them from /.zfs/snapshot/...
Others confirmed this bug: https://forum.proxmox.com/threads/zfs-snapshot-problem.120172/
The issue reported in the last comment seems like a "regression" introduced in zfs 2.1.7 (commit 4d22befde60087cbc6174122863353903df1d935 ) - at least on Debian(derived distros) with initramfs-tools:
- switching from
busybox mounttomount.zfscorrectly parses mountoptions for the root dataset, but additionally addsmntpoint=/rootto the mount options. This option is then prepended to the snapshot-mountpoint in.zfs/snapshot: https://github.com/openzfs/zfs/blob/a0105f6cd4a663167ebd80684abfd473adb35b93/module/os/linux/zfs/zfs_ctldir.c#L1100 /rootis the directory on Debian with initramfs-tools where the root-filesystem is mounted beforepivot_root(I assume thatnew_rootis similar for Arch andsysrootfor dracut). output in/proc/spl/kstat/zfs/dbgmsgwhen trying to access a snapshot in.zfs/snapshoton the/dataset:
1673286946 zfs_ctldir.c:1105:zfsctl_snapshot_mount(): Unable to automount /root/.zfs/snapshot/backup2 error=512
1673286946 zfs_ctldir.c:1105:zfsctl_snapshot_mount(): Unable to automount /root/.zfs/snapshot/backup2 error=512
- the issue did not occur with zfs < 2.1.7 on systems using
busybox mountin initramfs, becausebusybox mountdoes not callmount.zfs(but rather callsmount(2)quite directly) - without setting themntpointoption.
A possible workaround might be to allow mount.zfs to provide the mntpoint hint explicitly or an option to set it to the mountpoint property of the dataset (for use in initramfs/dracut).
A very superficial grep through the code makes it seem that vfs_mntpoint is only used in this context?
I the same problem, but with /var/lib/.zfs, and the sysroot workaround does not help:
[root@hel1-a:/]# mount -o bind /var/lib /sysroot
[root@hel1-a:/]# find sysroot/.zfs/snapshot/
sysroot/.zfs/snapshot/
sysroot/.zfs/snapshot/autosnap_2023-01-10_00:00:02_daily
sysroot/.zfs/snapshot/autosnap_2023-01-12_17:00:01_hourly
...
Same -- empty directories -- in /var/lib/.zfs/snapshot/<...>/.
I think you'd need to bind mount /var/lib to /sysroot/var/lib in this case (without explicitly trying)