efsw
efsw copied to clipboard
`fseventsd` “too many clients in system” error
I ran into a situation where efsw
was silently failing to notice filesystem events on my Mac and I couldn't figure out why. After lots of troubleshooting, I spotted this line in Console.app:
error 17:33:24.577894-0700 fseventsd too many clients in system (limit 1024)
I can find very, very little information about this, but it does seem to be true that fseventsd
enforces a hard system-wide limit on clients. Some experiments seemed to confirm that every individual directory added via addWatch
counts as a “client” for these purposes — there's a 1:1 correlation between calls to addWatch
and calls to FSEventStreamCreate
.
The error wouldn't actually happen until you called FSEventStreamStart
— efsw
doesn't check its return value, but it can return false
for opaque reasons:
It ought to always succeed, but in the event it does not then your code should fall back to performing recursive scans of the directories of interest as appropriate.
My use case is a Node module that wraps efsw
and exposes a file-watcher API in JavaScript. It's replacing an older library that did manual filesystem watching. In a single session, as many as 100 directories might be watched, and multiple sessions can run at once. (It's used by an Electron code editor that can have any number of windows open.) So it's extremely plausible for us to run up against this limit of 1024 clients on our own.
Our immediate workaround for this problem is to employ strategies to encourage watcher reuse and make it possible for several “wrapped” watchers to use the same underlying “native” (efsw
) watcher:
- a wrapped watcher for
/foo/bar/baz/
can reuse an native watcher on/foo/bar
if the latter already exists - in the reverse scenario — a new wrapped watcher on
/foo/bar
when the/foo/bar/baz
watcher already exists — creating a new native watcher at/foo/bar
, pointing both wrapped watchers at it, and destroying the/foo/bar/baz
native watcher - when a native watcher already exists at
/foo/bar/zort
, adding a wrapped watcher at/foo/bar/baz
triggers a new native watcher at/foo/bar
that can supply events for both paths and destroys the/foo/bar/zort
native watcher
This works well enough, but it increases the complexity of the implementation quite a bit. Our watchers don't need to be recursive, but this reuse strategy means that each watcher must be initialized as a recursive watcher (in case it is reused later). Since the reuse logic lives in the JavaScript, it's also the JavaScript code's job to receive all filesystem events and decide which ones match up with active watchers.
It's also possible to reuse too much — after all, we could create one watcher at the volume root and have all of our wrapped watchers use it, but we'd be “drinking from the firehose” and asking our wrapped watchers to spend lots of time processing events that they will eventually ignore (because they happen in directories that aren't being watched).
The quickest fix here, I think, would be to check for a false
return value from FSEventStreamStart
and handle it by falling back to a generic watcher (or the kqueue
watcher, perhaps). But that would be a pretty shallow fix.
The more thorough way to fix this would be to reduce the number of FSEventStreamRef
s created. The documentation for FSEventStreamCreate
suggests that there is no built-in limit to the number of paths that can be watched by a single stream; but research suggests that you'd have to restart the stream every time you change the list of watched paths.
I don't have the C++ experience to implement this, but imagine:
- Each
efsw::FileWatcher
aims to manage two differentFSEventStreamRef
s: one for paths that need recursion and one for paths that do not need recursion - The first time
watch
is called (or, if it’s called before any paths have been added, the first timeaddWatch
is called), one or both of theseFSEventStreamRef
s is created - Subsequent calls to
addWatch
work as follows:- The existing stream (recursive or non-recursive as needed) is stopped
- The new path is added to an internal array
- A new
FSEventStreamRef
is created with the new list of paths and is started - I think it would be possible to do this in
FSEvents
in such a way that no filesystem events are lost during the handover; but if I'm wrong, then the swap could instead be staggered, with the new stream starting before the old one is stopped
- The hard part here is issuing
efsw::WatchID
s and figuring out how to match up filesystem events to the watchers that initiated them; easier for the non-recursive watchers (each event belongs to exactly oneefsw::WatchID
) than the recursive watchers (each event can belong to multipleefsw::WatchID
s) - Calls to
removeWatch
would behave similarly to calls toaddWatch
— create a new stream without the removed path, then swap it in.