openzfs
No way to access snapshots within the `.zfs/snapshot` virtual folder from Samba share mounted via MacOS 14.6.1
System information
| Type | Version/Name |
|---|---|
| Distribution Name | Ubuntu |
| Distribution Version | 24.04 |
| Kernel Version | 6.8.0-39-generic |
| Architecture | amd64 |
| OpenZFS Version | zfs-2.2.2-0ubuntu9 |
Describe the problem you're observing
No way to access snapshots within the .zfs/snapshot virtual folder from Samba share mounted via MacOS 14.6.1
Describe how to reproduce the problem
Mount Samba share and try to browse mount at .zfs/snapshot. This used to work without setting snapdir=visible before an upgrade to MacOS 14.6 and Ubuntu 24.04.
Include any warning/errors/backtraces from the system logs
Attempts to access from within the .zfs/snaphot directory , even with sudo, gives errors like:
% ls -al
ls: snap_2024-07-24-16÷02÷10_HomeSync: No such file or directory
ls: snap_2024-07-25-03÷05÷53_HomeSync: No such file or directory
ls: snap_2024-07-25-16÷00÷55_HomeSync: No such file or directory
ls: snap_2024-07-25-16÷57÷58_HomeSync: No such file or directory
ls: snap_2024-07-26-16÷06÷29_HomeSync: No such file or directory
ls: snap_2024-07-27-16÷49÷33_HomeSync: No such file or directory
ls: snap_2024-07-28-15÷53÷12_HomeSync: No such file or directory
ls: snap_2024-07-28-16÷00÷15_HomeSync: No such file or directory
ls: snap_2024-07-29-14÷05÷12_HomeSync: No such file or directory
ls: snap_2024-07-29-14÷05÷52_HomeSync: No such file or directory
ls: snap_2024-07-29-14÷12÷29_HomeSync: No such file or directory
ls: snap_2024-07-29-14÷14÷47_HomeSync: No such file or directory
ls: snap_2024-07-29-14÷16÷31_HomeSync: No such file or directory
ls: snap_2024-07-29-14÷19÷44_HomeSync: No such file or directory
ls: snap_2024-07-29-14÷19÷55_HomeSync: No such file or directory
ls: snap_2024-07-29-14÷20÷42_HomeSync: No such file or directory
ls: snap_2024-07-29-14÷20÷57_HomeSync: No such file or directory
ls: snap_2024-07-29-14÷27÷19_HomeSync: No such file or directory
ls: snap_2024-07-29-14÷35÷18_HomeSync: No such file or directory
ls: snap_2024-07-29-16÷00÷07_HomeSync: No such file or directory
ls: snap_2024-07-30-13÷19÷01_HomeSync: No such file or directory
ls: snap_2024-07-30-17÷15÷54_HomeSync: No such file or directory
ls: snap_2024-07-30-22÷43÷18_HomeSync: No such file or directory
ls: snap_2024-07-31-16÷01÷02_HomeSync: No such file or directory
ls: snap_2024-08-01-16÷11÷34_HomeSync: No such file or directory
total 64
drwxrwxrwx 1 kimono staff 16384 Aug 6 23:45 .
drwxrwxrwx 1 kimono staff 16384 Aug 3 00:44 ..
ls: fts_read: No such file or directory
smb.conf contains:
zfsacl:expose_snapdir = True
veto files = /.windows/.mac/
When dataset is zfs set snapdir=visible ..., then most directories are visible but contents are not:
ls: snap_2024-07-29-14÷19÷55_HomeSync: No such file or directory
ls: snap_2024-07-30-17÷15÷54_HomeSync: No such file or directory
total 800
drwxrwxrwx 1 kimono staff 16384 Aug 6 23:45 .
drwxrwxrwx 1 kimono staff 16384 Aug 3 00:44 ..
drwxrwxrwx 1 kimono staff 16384 Jul 24 16:02 snap_2024-07-24-16÷02÷10_HomeSync
drwxrwxrwx 1 kimono staff 16384 Jul 25 03:05 snap_2024-07-25-03÷05÷53_HomeSync
drwxrwxrwx 1 kimono staff 16384 Jul 25 16:00 snap_2024-07-25-16÷00÷55_HomeSync
drwxrwxrwx 1 kimono staff 16384 Jul 25 16:57 snap_2024-07-25-16÷57÷58_HomeSync
drwxrwxrwx 1 kimono staff 16384 Jul 26 16:06 snap_2024-07-26-16÷06÷29_HomeSync
drwxrwxrwx 1 kimono staff 16384 Jul 27 16:49 snap_2024-07-27-16÷49÷33_HomeSync
drwxrwxrwx 1 kimono staff 16384 Jul 28 15:53 snap_2024-07-28-15÷53÷12_HomeSync
drwxrwxrwx 1 kimono staff 16384 Jul 28 16:00 snap_2024-07-28-16÷00÷15_HomeSync
drwxrwxrwx 1 kimono staff 16384 Jul 29 14:05 snap_2024-07-29-14÷05÷12_HomeSync
drwxrwxrwx 1 kimono staff 16384 Jul 29 14:05 snap_2024-07-29-14÷05÷52_HomeSync
drwxrwxrwx 1 kimono staff 16384 Jul 29 14:12 snap_2024-07-29-14÷12÷29_HomeSync
drwxrwxrwx 1 kimono staff 16384 Jul 29 14:14 snap_2024-07-29-14÷14÷47_HomeSync
drwxrwxrwx 1 kimono staff 16384 Jul 29 14:16 snap_2024-07-29-14÷16÷31_HomeSync
drwxrwxrwx 1 kimono staff 16384 Jul 29 14:19 snap_2024-07-29-14÷19÷44_HomeSync
drwxrwxrwx 1 kimono staff 16384 Jul 29 14:20 snap_2024-07-29-14÷20÷42_HomeSync
drwxrwxrwx 1 kimono staff 16384 Jul 29 14:20 snap_2024-07-29-14÷20÷57_HomeSync
drwxrwxrwx 1 kimono staff 16384 Jul 29 14:27 snap_2024-07-29-14÷27÷19_HomeSync
drwxrwxrwx 1 kimono staff 16384 Jul 29 14:35 snap_2024-07-29-14÷35÷18_HomeSync
drwxrwxrwx 1 kimono staff 16384 Jul 29 16:00 snap_2024-07-29-16÷00÷07_HomeSync
drwxrwxrwx 1 kimono staff 16384 Jul 30 13:19 snap_2024-07-30-13÷19÷01_HomeSync
drwxrwxrwx 1 kimono staff 16384 Jul 30 22:43 snap_2024-07-30-22÷43÷18_HomeSync
drwxrwxrwx 1 kimono staff 16384 Jul 31 16:01 snap_2024-07-31-16÷01÷02_HomeSync
drwxrwxrwx 1 kimono staff 16384 Aug 1 16:11 snap_2024-08-01-16÷11÷34_HomeSync
It seems as though ZFS no longer auto-mounts snapshots when a request is made from a SMB client.
However, if I stir the snapshots on the host, and let the auto-mounter work, the guest system now shows the contents of re: ls -alR but not re: a normal statx!
I'll start by saying that OpenZFS 2.2.2 never had support for kernel 6.8; Ubuntu backported experimental patches that we hadn't completed, and I don't know what they've done since. This is not to blame them, but rather to say, this isn't a combination we technically support, and it might just be busted. But let's assume the best for the moment.
You say it worked before, so what actually changed? You mention client and OS; I suppose the latter means you got new kernel, OpenZFS and Samba? What versions of each did you have before?
It's not clear to me if you're saying that it's working if you do it directly on the host? I don't know what you're doing in the last paragraph ("stir the snapshots").
If Samba is doing something different than it used to, then that might be it. If it's doing the same thing then yes, perhaps a response has changed. You mention statx, how exactly has its response changed?
It might be a real bug since fixed, but I didn't quickly find a PR that sounded like it. Could easily mean I missed something.
Basically, divide and conquer. See if you can narrow it down!
I'll start by saying that OpenZFS 2.2.2 never had support for kernel 6.8; Ubuntu backported experimental patches that we hadn't completed, and I don't know what they've done since.
Understood. Hope no offense is taken. I see a thumbs down emoji! It seemed like a possible bug in the auto-mounter, like perhaps it should mount on a larger variety of system calls.
You say it worked before, so what actually changed? You mention client and OS; I suppose the latter means you got new kernel, OpenZFS and Samba? What versions of each did you have before?
I upgraded from Ubuntu 22.04 to 24.04 while using MacOS 13, except it required the use of the zfsacl:expose_snapdir = True setting in the smb.conf which it didn't previously.
This config seemed to work fine until an update to MacOS 14+.
It's not clear to me if you're saying that it's working if you do it directly on the host? I don't know what you're doing in the last paragraph ("stir the snapshots").
On the host, I get the auto-mouter to mount the snapshots. Like -- if I do a read_dir on the .zfs/snapshot folder to give me all the snapshot directories, and statx on a file (in this case .zshrc) in each snapshot directory, by joining the snapshot directories with the relative path, via the httm command.
This causes the auto-mounter to mount all snapshot directories.
If Samba is doing something different than it used to, then that might be it. If it's doing the same thing then yes, perhaps a response has changed. You mention statx, how exactly has its response changed?
Yes, it could be. Samba version is: Version 4.19.5-Ubuntu.
Basically I'd like to be able to do what I did previously: On the client, I'd like to do a read_dir on the .zfs/snapshot folder on the client, and statx on a file (in this case .zshrc) in each snapshot directory, via httm, and have these requests auto-mount the snapshot directories.
It might be a real bug since fixed, but I didn't quickly find a PR that sounded like it. Could easily mean I missed something.
I searched the issues for something similar but didn't discover anything.
Okay I have a workaround: If I do an opendir and read_dir on the snapshot directory of the file to be statx-ed and iterate over at least one entry, before I fstat that file, I can now fstat each file. Of course that's kinda bizarre, and very slow, but it works.
Also -- I think smbd is using fstat instead of statx. And I think that may be the difference? Could it be the auto-mounter is not mounting on fstat calls? That kind of makes sense if perhaps the auto-mounter is looking for paths not fds.
In any event, this perfectly describes my problem and the ask. I'd like snapshots to auto-mount when any stat type call is made instead of having to wait to complete the opendir into read_dir iterator. This was once the behavior of Samba + ZFS but it is apparently not any longer. It makes sense to do this because it is much faster than waiting 10-100x the time for the other system calls to complete. It's also more correct. When I stat a file in a snapshot directory, it should show up. We shouldn't have to enter that directory first.
FYI to confirm the source of this behavior seems to be Samba on the Linux server. See:
Starting with version 4.14 Samba provides core infrastructure code that allows basing all access to the server's filesystem on file handles and not on paths. An example of this is using fstat() instead of stat(), or SMB_VFS_FSTAT() instead of SMB_VFS_STAT() in Samba parlance.
From: https://wiki.samba.org/index.php/The_New_VFS
@robn would you advise?
This is good info, thank you. Unfortunately I don't know anything much about Samba; my questions were mostly for triage.
The next thing I suppose is to see if you can reproduce it on a stock OpenZFS., to rule in/out Ubuntu's version. You could possibly even just ask Ubuntu, since they should be supporting their whole stack.
Alternately, a simple reproduction script or program that doesn't require installing Samba or Ubuntu. We'll need to be able to reproduce it to fix it if there is a bug anyway, and if it doesn't reproduce it on stock OpenZFS, then you'll have narrowed it down.
Minimal reproducer shows my hypothesis might not be the cause:
use std::fs::OpenOptions;
use std::os::fd::AsRawFd;
use std::path::PathBuf;
use std::thread::sleep;
use std::time::Duration;
pub type GenericResult<T> = Result<T, Box<dyn std::error::Error + Send + Sync>>;
fn main() -> GenericResult<()> {
let path = PathBuf::from("/.zfs/snapshot/autosnap_2024-06-01_00:00:29_monthly/etc/passwd");
let path_md = nix::sys::stat::stat(&path).ok();
eprintln!("{:?}", path_md);
let file = OpenOptions::new()
.read(true)
.open("/home/kimono/.zfs/snapshot/autosnap_2024-08-11_10:05:28_hourly/.zshrc")?;
sleep(Duration::from_secs(30));
let fd = file.as_raw_fd();
let file_md = nix::sys::stat::fstat(fd).ok();
eprintln!("{:?}", file_md);
Ok(())
}
I have seem to hit the same issue. What I did before was:
- mount the
.zfs/snapshotdir somehwere and then share this in samba - [windows] users that mounted that share saw all the snapshots and could enter each snapshot and see the files in there
At some point this changed:
- now the windows user still see the snapshots but when they enter the snapshot, no content is listed.
What I tried also:
- through cli on the samba server enter that snapshot and list the contents (which works...)
- then try to access that same snapshot through windows mounted smb but it still did not list any files.
Because of this thread I added now zfsacl:expose_snapdir = True also. This allows me in "normal" samba shares to see the .zfs/snapshot folder. I can enter them on windows and snapshots are listed. But upon entering a snapshot still nothing listed.
So, I did some testing in a vm. I did mount:
tankNixos/data to /data
and then I did bind mount
/data/.zfs/snapshot to /samba
In the hardware-configuration.nix:
fileSystems."/data" =
{ device = "tankNixos/data";
fsType = "zfs";
};
fileSystems."/samba" =
{ device = "/data/.zfs/snapshot";
options = [ "bind" ];
};
I did then go through all NixOS releases sind 2020. The last one ist release 24.11 which uses zfs-2.2.5-1 and samba 4.20.4. There it still works. I can access the /samba share through smb with windows, see the snapshots, enter them and have the contents listed.
However when I switched to NixOS Unstable, which uses zfs-2.3.0-1 and samba 4.20.4 it doesn't work anymore. It still lists the snapshots but upon entering no contents are being listed.
I narrowed it now further down. On zfs-2.2.6 it still works but on zfs-2.2.7 it stops working.
I wonder if the change introduced in: https://github.com/openzfs/zfs/pull/16833 could be the culprint?
I narrowed it further down. Thank NixOS I can easily change what zfs source should be used.
First I compiled a list of commits between 2.2.6 and 2.2.7 releases.
Then I cloned the zfs repository and run:
git log "--pretty=%h %D" baa5031..e269af1 > /tmp/zfs_commits.txt
That returned around 220 commits. So I started narrowing it down, selecting a commit around the middle. Check if samba behaviour was as expected or not... then chose another commit around the middle etc... always narrowing down.
So in the end I found that with commit 63fbbe871 samba still shows the content in the snapshots but with commit 308d04ac3 it doesn't show anymore.
@usaleem-ix
Maybe change the title? It doesn't only affect MacOS
Pleased to change. What do you suggest?
Maybe just: No way to access snapshots within the .zfs/snapshot virtual folder from Samba share Because it doesn't work on MacOS anymore and neither on linux and I assume it won't work on BSD also (but don't know about that) Or:
No way to access snapshots within the .zfs/snapshot virtual folder from Samba share via MacOS 14.6.1 and Linux