How to properly implement filesystem notifications (fsnotify)

[markg85]

trafficstars

Hi,

I'm working on a filesystem implementation with this, rather nice, library!

The thing i'm stuck on is the concept of how to properly implement fsnotify support. When i look at the two [1, 2] provided examples then the current way seems to be:

• Run a filesystem update loop in a set interval (1 second)
• Send filesystem updates in that loop

When you have a file that changes every second (like the examples demonstrate) then this logic probably works just fine. But that's not a realistic scenario.

What i did find out, or i'm wrong, is that fsnotify events should be send outside the code path of where you create/update/delete files themselves to prevent deadlocks. So executing a create call immediately followed by an fsnotify notifier.inval_entry (notifier comes from the Session object) is a big no-no. Thus a serial approach of create -> notify event does not seem to be an option.

This makes me think/assume that i should:

• Build a queue of filesystem events
• Have a separate thread where that queue is read and send to the kernel

Is that the intended approach of fsnotify support? If not, what is the intended approach?

For reference, i'm making a filesystem that essentially exposes external data (think of a web api) to a local filesystem. I'm receiving events from that external source which i need to map to fsnotify events.

[1] https://github.com/cberner/fuser/blob/master/examples/notify_inval_entry.rs [2] https://github.com/cberner/fuser/blob/master/examples/notify_inval_inode.rs

[rarensu]

I managed to implement your no-no solution. It works great, until it hangs my filesystem and then shortly after, my whole desktop.

I used Arc<RwLock<Option<fuser::Notifier>>> to move the notifier from the session into the filesystem loop.

[markg85]

I managed to implement your no-no solution. It works great, until it hangs my filesystem and then shortly after, my whole desktop.

I used Arc<RwLock<Option<fuser::Notifier>>> to move the notifier from the session into the filesystem loop.

Ha 😆 That made me laugh! Guess my gut feeling of it being a no-no is indeed a no-no ;) Would you mind sharing a diff of your work, that would be super neat!

[rarensu]

I was testing my file system and I found that if I did a rename operation followed immediately by query to see the result of the rename operation then I would be seeing cached results which were incorrect. That is why I was motivated to try to make the notify action occur as soon as possible after the rename action. I suspect i need to put the notify action behind something like an async so that the session thread can continue receiving file system requests while the notify is pending. I kind of want to avoid creating a notify queue structure if I can. That sounds like it will get even more complex in the end when we migrate to multi threaded session. I work in an HPC environment where spawning lots of threads is not a problem, but waiting for a queue to empty is a big problem. The optimal solution for your needs might be different.

Cargo.toml

[dependencies]
fuser = { version="0.15", features=["abi-7-15"] }

Rust code

use fuser::{..., Notifier};
use std::sync::{Arc,RwLock};
use log::info;

const TTL: Duration = Duration::from_secs(1);
pub struct My_FS {
    ..., 
    notifier: Arc<RwLock<Option<Notifier>>>,
}
impl My_FS {
    pub fn new(..., notifier: Arc<RwLock<Option<Notifier>>>) -> Self {
        My_FS {
            ...,
            notifier,
        }
    }
    ...
}
impl Filesystem for My_FS {
    fn rename(...) {
        ...
        if let Some(notifier) = self.notifier.read().unwrap().as_ref() {
            info!("My_FS::rename: Notifying kernel of invalid cache entries.");
            // TODO async this
            if let Err(err) = notifier.inval_entry(parent, name) {
                error!("Failed to invalidate entry for parent {}: {:?}", parent, err);
            }
            // TODO async this
            if let Err(err) = notifier.inval_entry(newparent, newname) {
                error!("Failed to invalidate entry for newparent {}: {:?}", newparent, err);
            }
        } else {
            info!("My_FS::rename: Skip notify; Notifier not set.");
        }
    }

...
}

fn main() -> Result<(), i32> {
    env_logger::init();
    ...
    let mountpoint ...;
    let options ...;
    let notifier: Arc<RwLock<Option<Notifier>>> = Arc::new(RwLock::new(None));
    let fs = My_FS::new(..., Arc::clone(&notifier));
    let mut this_session = fuser::Session::new(fs, &mountpoint, &options).unwrap();

    // Update the notifier after the session is created
    {
        let mut notifier_lock = notifier.write().unwrap();
        *notifier_lock = Some(this_session.notifier());
    }

    // wait for the session to finish
    // TODO multithread
    this_session.run().unwrap();

    Ok(())
}

[markg85]

Thank you so much @rarensu !

While the needs indeed might be different, this helps a lot!

FUSE and file system notifications are one heck of a complicated beast in the Linux world (and frankly, Rust isn't making that any easier) so your example is a great to build upon.

[rarensu]

I wouldn't say that rust is making this easier; I don't think rust really ever makes anything easier. But, I've been working on this file system for a few weeks, and this is literally the first time it ever crashed due to stress. You can bet if I was writing a file system in an unsafe language, it would have crashed every. single. day.

[markg85]

I wouldn't say that rust is making this easier

😉 I said: Rust isn't making that any easier

But if you look at other languages then Rust does certainly have an appeal for filesystems here. That's only for it's concepts. The language itself is.. well.. let's not go into that.

I'm tempted to go for C++ instead but that's because i'm working on a cross platform filesystem where it might be easier to do this in C++. Especially as i would have to interact with libraries from Mac/Windows which already would mean using FFI in Rust (and therefore making things soooo much more complicated). Alternatively i might want to fork this very project and implement those mac/windows bits in here. I'm not set on any of this yet, it's very much a work in progress.