Provide hint when running out of watches / instances; leak via react.py

[blueyed]

I've seen the following errors (via Django):

ERROR:pyinotify:add_watch: cannot watch /path/to/file/ WD=-1, Errno=No space left on device (ENOSPC)

This can be caused when running out of inotify instances or watches (ref).

I could track it down to react causing it when it is reloading, at least the number of instances gets increased then.

I've not taken a closer look yet, but it's probably rather trivial for you to spot it: https://github.com/copton/react/blob/master/react.py

Anyway, the issue with pyinotify is that the exception / error message could be a bit more informative maybe, instead of only providing the ENOSPC reference.

The limits that might be relevant are:

% for i in /proc/sys/fs/inotify/*; do echo "${i:t}: $(<$i)"; done
max_queued_events: 16384
max_user_instances: 128
max_user_watches: 524288

Maybe pyinotify can help in use cases like this, too, by providing some garbarge collection / cleanup automatically?

[blueyed]

It turns out that notifier.stop() needs to be called on reload: https://github.com/copton/react/pull/1/files

Would it be possible for pyinotify to handle this, when the old notifier reference gets overwritten / garbage collected?

[timabbott]

Is there any chance of this leak issue being resolved? There's a lot of issues around the Internet resulting from pyinotify both leaking watched files and also giving this confusing ENOSPC error message. See e.g. https://code.djangoproject.com/ticket/23523 or https://github.com/zulip/zulip/issues/12565.

[blueyed]

@timabbott can you try https://github.com/seb-m/pyinotify/pull/96?

[timabbott]

I'll try to test that out; because I don't have a consistent reproducer (it starts happening after a few days of use), I'm not sure I'll be able to have a clear "this fixed it" or not story until after a few weeks of use.