efsw icon indicating copy to clipboard operation
efsw copied to clipboard

`fseventsd` “too many clients in system” error

Open savetheclocktower opened this issue 4 months ago • 3 comments

I ran into a situation where efsw was silently failing to notice filesystem events on my Mac and I couldn't figure out why. After lots of troubleshooting, I spotted this line in Console.app:

error	17:33:24.577894-0700	fseventsd	too many clients in system (limit 1024)

I can find very, very little information about this, but it does seem to be true that fseventsd enforces a hard system-wide limit on clients. Some experiments seemed to confirm that every individual directory added via addWatch counts as a “client” for these purposes — there's a 1:1 correlation between calls to addWatch and calls to FSEventStreamCreate.

The error wouldn't actually happen until you called FSEventStreamStartefsw doesn't check its return value, but it can return false for opaque reasons:

It ought to always succeed, but in the event it does not then your code should fall back to performing recursive scans of the directories of interest as appropriate.

My use case is a Node module that wraps efsw and exposes a file-watcher API in JavaScript. It's replacing an older library that did manual filesystem watching. In a single session, as many as 100 directories might be watched, and multiple sessions can run at once. (It's used by an Electron code editor that can have any number of windows open.) So it's extremely plausible for us to run up against this limit of 1024 clients on our own.

Our immediate workaround for this problem is to employ strategies to encourage watcher reuse and make it possible for several “wrapped” watchers to use the same underlying “native” (efsw) watcher:

  • a wrapped watcher for /foo/bar/baz/ can reuse an native watcher on /foo/bar if the latter already exists
  • in the reverse scenario — a new wrapped watcher on /foo/bar when the /foo/bar/baz watcher already exists — creating a new native watcher at /foo/bar, pointing both wrapped watchers at it, and destroying the /foo/bar/baz native watcher
  • when a native watcher already exists at /foo/bar/zort, adding a wrapped watcher at /foo/bar/baz triggers a new native watcher at /foo/bar that can supply events for both paths and destroys the /foo/bar/zort native watcher

This works well enough, but it increases the complexity of the implementation quite a bit. Our watchers don't need to be recursive, but this reuse strategy means that each watcher must be initialized as a recursive watcher (in case it is reused later). Since the reuse logic lives in the JavaScript, it's also the JavaScript code's job to receive all filesystem events and decide which ones match up with active watchers.

It's also possible to reuse too much — after all, we could create one watcher at the volume root and have all of our wrapped watchers use it, but we'd be “drinking from the firehose” and asking our wrapped watchers to spend lots of time processing events that they will eventually ignore (because they happen in directories that aren't being watched).


The quickest fix here, I think, would be to check for a false return value from FSEventStreamStart and handle it by falling back to a generic watcher (or the kqueue watcher, perhaps). But that would be a pretty shallow fix.


The more thorough way to fix this would be to reduce the number of FSEventStreamRefs created. The documentation for FSEventStreamCreate suggests that there is no built-in limit to the number of paths that can be watched by a single stream; but research suggests that you'd have to restart the stream every time you change the list of watched paths.

I don't have the C++ experience to implement this, but imagine:

  • Each efsw::FileWatcher aims to manage two different FSEventStreamRefs: one for paths that need recursion and one for paths that do not need recursion
  • The first time watch is called (or, if it’s called before any paths have been added, the first time addWatch is called), one or both of theseFSEventStreamRefs is created
  • Subsequent calls to addWatch work as follows:
    • The existing stream (recursive or non-recursive as needed) is stopped
    • The new path is added to an internal array
    • A new FSEventStreamRef is created with the new list of paths and is started
    • I think it would be possible to do this in FSEvents in such a way that no filesystem events are lost during the handover; but if I'm wrong, then the swap could instead be staggered, with the new stream starting before the old one is stopped
  • The hard part here is issuing efsw::WatchIDs and figuring out how to match up filesystem events to the watchers that initiated them; easier for the non-recursive watchers (each event belongs to exactly one efsw::WatchID) than the recursive watchers (each event can belong to multiple efsw::WatchIDs)
  • Calls to removeWatch would behave similarly to calls to addWatch — create a new stream without the removed path, then swap it in.

savetheclocktower avatar Oct 27 '24 20:10 savetheclocktower