.zfs/snapshot/ directories almost, but not quite, gone after being destroyed

[rincebrain]

(Given my luck with not finding prior cases, I'm not hopeful, but I searched a number of keywords and found a number of odd behaviors of .zfs/snapshot in issues, but none seemed to be this one.)

System information

Type	Version/Name
Distribution Name	Debian
Distribution Version	10.10
Kernel Version	4.19.0-17-amd64
Architecture	x86_64
OpenZFS Version	v2.0.3-8~bpo10+1

Describe the problem you're observing

I was automating replicating the contents of individual snapshots onto a new dataset without using send|recv, and I ran into a strange case.

I was doing something like

for SNAP in snaplist; do if [ ! -e DST/.zfs/snapshot/$SNAP ]; then rsync -a --delete-after SRC/.zfs/snapshot/$SNAP/ DST/ && zfs snapshot DST@SNAP;fi;done;

and to my surprise, I was winding up with it sometimes destroying all the files on DST and then syncing them anew.

It turns out what was happening was that, if SNAP was mounted in .zfs/snapshot/SNAP, automatic snapshot curation was destroying SNAP between list time and sync time, and that led to the following exciting behavior:

# ls SRC/.zfs/snapshot/DELETEDSNAP/   && echo "ONFIRE"
ONFIRE
# ls SRC/.zfs/snapshot/DELETEDSNAP/.   && echo "ONFIRE"
ls: cannot access 'SRC/.zfs/snapshot/DELETEDSNAP/.': Object is remote
# rsync -avn --inplace --delete-after SRC/.zfs/snapshot/DELETEDSNAP/ /tmp/dummy/ && echo "EVERYTHING'S FINE";
building file list ... done
./
deleting c
deleting b
deleting a

sent 54 bytes  received 18 bytes  144.00 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)
EVERYTHING'S FINE
# mount | grep SNAP
#

Whoops.

There are obviously any number of ways I can avoid this - checking for errors in existence on both sides before syncing, checking for errors from the above stat call before same, using zfs hold...but it was still pretty surprising, to me, so I figured I'd ask and see if this is expected but counterintuitive or a bug, and at least give other people who hit it something to find.

Include any warning/errors/backtraces from the system logs

Well, sometimes my comment in #11632 happens a while after one of the aforementioned reproducing runs. So that's "great".

[stale[bot]]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.