System information

Type	Version/Name
Distribution Name	Ubuntu 20.04 LTS - Pi4 Edition
Distribution Version	Ubuntu 20.04 LTS - Pi4 Edition
Linux Kernel	5.4.0-1008-raspi
Architecture	arm64
ZFS Version	0.8.3-1ubuntu12
SPL Version	0.8.3-1ubuntu12

Installed using a guide I wrote (https://github.com/jrcichra/rpi-zfs-root), which involves apt install zfs-dkms, zfsutils-linux, and zfs-initramfs.

apt install zfs-dkms did some building

Describe the problem you're observing

zfs list

ubuntu@justinpi:/home/.zfs/snapshot$ zfs list
NAME                                  USED  AVAIL     REFER  MOUNTPOINT
rastank                                     2.26G  25.8G       96K  /mnt/rastank
rastank/ubuntu                              2.24G  25.8G     1.09G  /
rastank/ubuntu/home                       51.6M  25.8G      164K  /home
rastank/ubuntu/home/lancache               51.0M  25.8G     51.0M  /home/ubuntu/lancache
rastank/ubuntu/var                          1.08G  25.8G     76.8M  /var
rastank/ubuntu/var/lib                     825M  25.8G      295M  /var/lib
rastank/ubuntu/var/lib/docker              517M  25.8G      101M  /var/lib/docker
rastank/ubuntu/var/log                   19.3M  25.8G     3.71M  /var/log
rastank/ubuntu/var/spool                 176K  25.8G      120K  /var/spool
ubuntu@justinpi:/home/.zfs/snapshot$

.zfs/snapshot directories are not mounting in the proper location (this could totally be a zfs noob error)

ubuntu@justinpi:/home/.zfs/snapshot$ cd install/
-bash: cd: install/: Too many levels of symbolic links
ubuntu@justinpi:/home/.zfs/snapshot$

ubuntu@justinpi:/home/.zfs/snapshot$ mount | grep zfs
rastank/ubuntu on / type zfs (rw,relatime,xattr,noacl)
rastank/ubuntu/home on /home type zfs (rw,relatime,xattr,noacl)
rastank/ubuntu/home/lancache on /home/ubuntu/lancache type zfs (rw,relatime,xattr,noacl)
rastank/ubuntu/var on /var type zfs (rw,relatime,xattr,noacl)
rastank/ubuntu/var/lib on /var/lib type zfs (rw,relatime,xattr,noacl)
rastank/ubuntu/var/lib/docker on /var/lib/docker type zfs (rw,relatime,xattr,noacl)
rastank/ubuntu/var/log on /var/log type zfs (rw,relatime,xattr,noacl)
rastank/ubuntu/var/spool on /var/spool type zfs (rw,relatime,xattr,noacl)
rastank/ubuntu/home@install on /.zfs/snapshot/install type zfs (ro,relatime,xattr,noacl)
ubuntu@justinpi:/home/.zfs/snapshot$

You can see for some reason it's mounting the snapshot under /.zfs, which is not what I was expecting. If I go there...

ubuntu@justinpi:/home/.zfs/snapshot$ cd /.zfs/snapshot/install/
ubuntu@justinpi:/.zfs/snapshot/install$ ls
ubuntu
ubuntu@justinpi:/.zfs/snapshot/install$

I'll see what I expected to appear right after I cd'd (/home with the ubuntu homedir in it)

Describe how to reproduce the problem

See my asciinema recording a better 'flow' understanding (or the wording above):

To fully reproduce the issue, follow my guide on a Raspberry Pi 4 root ZFS install. Make a snapshot of something not in /, then try to cd into the snapshot.

Include any warning/errors/backtraces from the system logs

Didn't see any errors in the common places (syslog was clean)

ubuntu@justinpi:~$ sudo cat /proc/spl/kstat/zfs/dbgmsg
timestamp    message 
188          spa.c:5638:spa_tryimport(): spa_tryimport: importing rastank
188          spa_misc.c:407:spa_load_note(): spa_load($import, config trusted): LOADING
188          vdev.c:124:vdev_dbgmsg(): disk vdev '/dev/mmcblk0p3': best uberblock found for spa $import. txg 18912
188          spa_misc.c:407:spa_load_note(): spa_load($import, config untrusted): using uberblock with txg=18912
188          spa_misc.c:407:spa_load_note(): spa_load($import, config trusted): LOADED
188          spa_misc.c:407:spa_load_note(): spa_load($import, config trusted): UNLOADING
188          spa.c:5490:spa_import(): spa_import: importing rastank
188          spa_misc.c:407:spa_load_note(): spa_load(rastank, config trusted): LOADING
188          vdev.c:124:vdev_dbgmsg(): disk vdev '/dev/mmcblk0p3': best uberblock found for spa rastank. txg 18912
188          spa_misc.c:407:spa_load_note(): spa_load(rastank, config untrusted): using uberblock with txg=18912
188          spa_misc.c:407:spa_load_note(): spa_load(rastank, config trusted): spa_load_verify found 0 metadata errors and 10 data errors
188          mmp.c:248:mmp_thread_start(): MMP thread started pool 'rastank' gethrtime 188884702554
193          spa.c:7592:spa_async_request(): spa=rastank async request task=1
193          spa_misc.c:407:spa_load_note(): spa_load(rastank, config trusted): LOADED
193          spa_history.c:316:spa_history_log_sync(): txg 18914 open pool version 5000; software version unknown; uts justinpi 5.4.0-1008-raspi #8-Ubuntu SMP Wed Apr 8 11:13:06 UTC 2020 aarch64
195          spa.c:7592:spa_async_request(): spa=rastank async request task=32
195          spa_history.c:316:spa_history_log_sync(): txg 18916 import pool version 5000; software version unknown; uts justinpi 5.4.0-1008-raspi #8-Ubuntu SMP Wed Apr 8 11:13:06 UTC 2020 aarch64
1585761823   spa_history.c:305:spa_history_log_sync(): command: zpool import -N rastank
1589852449   spa.c:7592:spa_async_request(): spa=rastank async request task=1
1589852455   spa_history.c:305:spa_history_log_sync(): command: zpool online -e rastank /dev/mmcblk0p3
1589853210   metaslab.c:2736:metaslab_condense(): condensing: txg 19027, msp[1] ffff0000e9894000, vdev id 0, spa rastank, smp size 16424, segments 515, forcing condense=FALSE
1589853257   metaslab.c:2736:metaslab_condense(): condensing: txg 19036, msp[8] ffff0000e6ce2800, vdev id 1, spa rastank, smp size 16488, segments 328, forcing condense=FALSE
1589854214   metaslab.c:2736:metaslab_condense(): condensing: txg 19223, msp[3] ffff0000e9890800, vdev id 0, spa rastank, smp size 16464, segments 148, forcing condense=FALSE

ubuntu@justinpi:~$ sudo zpool events
TIME                           CLASS
Dec 31 1969 19:03:13.835999884 sysevent.fs.zfs.history_event
Dec 31 1969 19:03:15.111999883 sysevent.fs.zfs.config_sync
Dec 31 1969 19:03:15.111999883 sysevent.fs.zfs.pool_import
Dec 31 1969 19:03:15.131999883 sysevent.fs.zfs.history_event
Dec 31 1969 19:03:15.343999883 sysevent.fs.zfs.config_sync
May 18 2020 21:40:49.982041996 sysevent.fs.zfs.config_sync
May 18 2020 21:40:49.986041996 sysevent.fs.zfs.config_sync
ubuntu@justinpi:~$

May 19 '20 02:05 jrcichra

This is the same issue as https://github.com/openzfs/zfs/issues/9958, however the difference here is that no bind mounts are needed to cause the issue. I've been experiencing this as well since at least 0.8.2.

May 26 '20 03:05 TheUbuntuGuy

I think this is a duplicate of #9461 which seems to have more activity.

Jun 14 '20 21:06 TheUbuntuGuy

There are a variety of cases which need to be considered, thanks for linking to those other issues for reference.

Jun 15 '20 20:06 behlendorf

~~I can reproduce this and for me it seems the root cause is datasets with 'mountpoint' inherited. When accessing their snapshot directory, it gets automounted under the first dataset with explicit mountpoint above instead.~~ Edit: dextarr's comment below seems to be more relevant, setting the mountpoint only helped temporarily.

Nov 22 '20 00:11 TPolzer

I can too... creating a dataset under an existing one (inherited mountpoint by default) and changing into a snapshot directory works only until a reboot (so probably export+import zpool but as im on zfs-root its hard to tell)

after the reboot it gets mounted to /.zfs instead of /blah/.zfs:

zpool/system/blah@zfs-auto-snap_frequent-2020-12-03-1130 on /.zfs/snapshot/zfs-auto-snap_frequent-2020-12-03-1130 type zfs (ro,relatime,xattr,posixacl)

OS: kde neon 20.04 fully patched (it was working fine on bionic)

Dec 03 '20 11:12 dextarr

i was trying to reproduce it in a vm with livecd and non-root zpool to no avail... could it be root zpool related?

Dec 03 '20 12:12 dextarr

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.

Dec 03 '21 15:12 stale[bot]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.

Aug 10 '23 03:08 stale[bot]

Bump for stale bot. Still experiencing this.

Oct 07 '23 23:10 tilgovi

Scratch that. I just upgraded to Ubuntu 23.10, which is currently shipping v2.2.0-rc3 with some cherry-picked kernel compatibility patches and a couple of bugfixes that look unrelated. I'm no longer experiencing this issue. It would be great if anyone using the vanilla v2.2.0 release candidates could check whether this is still happening.

Oct 08 '23 21:10 tilgovi

It would be great if anyone using the vanilla v2.2.0 release candidates could check whether this is still happening.

I'm still experiencing some variation of this even with zfs 2.2-git:

[niko@arch-phoenix ~]$ ls -al /.zfs/snapshot/zrepl_20231009_050257_000/
total 0
drwxrwxrwx  1 root root 0 Oct  9 07:02 .
drwxrwxrwx 39 root root 2 Oct  9 08:52 ..
[niko@arch-phoenix ~]$ mount | grep zrepl_20231009_050257_000
[niko@arch-phoenix ~]$

As you can see sometimes it doesn't automount snapshots when you try to access them.

Oct 09 '23 06:10 darkbasic

I don't know if the Debian packages (currently sid's 2.2.4-1) are 'vanilla' enough, but as I'm seeing a different bugs that is fairly serious and I'm not sure if it's a Debian problem or an upstream problem, I'll probably be taking the time to build from git in the near future anyway.