Blocking readFileAtPath in LoopbackFS delays next contentsOfDirectoryAtPath

[radj]

Experiment: I modified LoopbackFS to block for 10 seconds when readFileAtPath: is read for a specific file path to simulate latency from a source. I am on 10.13 with osxfuse 3.7.1 installed. This also occurs with 10.12.6.

Observed: When I open this file and immediately browse to another folder when readFileAtPath: starts to block, contentsOfDirectoryAtPath: does not get called immediately. Rather, contentsOfDirectoryAtPath: gets only called when readFileAtPath: unblocks after 10 seconds.

Desired: A blocking readFileAtPath: should not block other file/dir operations in the file system.

Question: I looked in the mount options wiki page and can't seem to find an option that allows this behavior. Is the desired behavior possible to achieve?

[gtimmy]

How do you test? With a Finder.app? If yes, try to reproduce it with a Terminal.app I faced with a similar issue with a Finder

[radj]

@gtimmy If using only one session with Terminal, it's not possible to reproduce because the read will have to wait to finish (ex cat) before you can call another command. It's not possible to test asynchronous behavior with Terminal unless forking or using multiple sessions.

How are you testing with Terminal?

[gtimmy]

Yes, I'm talking about multiple sessions.

[radj]

Gotcha. I'll try again when I get back to my desk.

[radj]

@gtimmy So I tried this setup: two separate Terminal session windows, both set working directory by cd into the LoopbackFS mounted volume.

1. On Terminal A, cat test-sleep-file.bin which triggers readFileAtPath:. A sleep(10) inside readFileAtPath: simulates a read wait.
2. On Terminal B, ls -la to trigger contentsOfDirectoryAtPath:. The ls on Terminal B blocks and waits for the readFileAtPath: to unblock.

So it still occurs with Terminal.

[raman-sidarakin-epam]

@gtimmy , @radj Hi, everyone. Faced the same behavior. When readFileAtPath: is blocked (waits for something), then all metadata callbacks like contentsOfDirectoryAtPath: or attributesOfItemAtPath: not being called until readFileAtPath: unblocked. It also reproducible with Loopback FS using Finder.app or Terminal. Did you find some workaround or some specific approach to avoid this?

[vadim-alexeev-distillery]

Hi all, I faced the same issue. All callbacks are blocked until readFileAtPath is finished. In case when readFileAtPath falls into loading data from the network this issue leads to the bad user experience. Finder may hang trying to get file attributes till the reading is done. For a big file, after several minutes hanging Fuse drive is finally unmounted.

@gtimmy, @radj have you achieved some progress concerning this? @bfleischer is there any chance to get this resolved in one of the future releases?

[jeremyspiegel]

I'm seeing the same problem. I ran the loopback filesystem, and introduced a 10 second sleep in the read() function. Then, in one terminal window, I ran cat on a file, and while that was hung, in another terminal window, I ran stat on the same file. The stat hung until the read had completed.

[jeremyspiegel]

I enabled FUSE_TRACE_OP in kext/osxfuse/fuse.h, with a 10 second sleep in the read() function in my loopbackfs. I also added an IOLog for right before fuse_vnop_read returns.

I found that fuse_vnop_getattr isn't called until after the 10-second pause when fuse_vnop_read returns. So it seems like the wait that is hanging the stat call might be in the kernel itself rather than in the osxfuse kext.

[jeremyspiegel]

I also tried setting a 10-second sleep at the top of fuse_vnop_read via msleep (rather than sleeping in my user-mode FS), and saw the same behavior where the fuse_vnop_getattr for the stat wasn't called until after the read returned.

[jeremyspiegel]

I was unclear about how nodes are locked, but now I see now that the node is locked before entering fuse_vnop_read in fuse_biglock_vnop_read. If I call fuse_nodelock_unlock(fvdat) before cluster_read in fuse_vnop_read, and then fuse_nodelock_lock(fvdat, FUSEFS_EXCLUSIVE_LOCK) afterward, then my stat is able to complete while the read is blocked.

I'm not sure if this is safe to do though, since in the other places I see fuse_nodelock_unlock called (fuse_vnop_mmap/fuse_vnop_open), there is a comment saying "Releasing the fusenode lock during a vnop is dangerous, but it is considered safe..." and then a reason is given for why it is safe in that case.

@bfleischer, thoughts?

[matiaskotlik]

Is this still a bug?