Wishlist: metric for inotify watches

[albertvaka]

Why?

It's increasingly common for programs to use Linux's inotify to watch files. Some apps leak the file descriptors of the inotify watches. When that happens, the machine will eventually run out of inotify watches.

What?

It would be useful to have a metric with the number of inotify watches to be able to monitor it.

[brian-brazil]

Do you know how to obtain this from /proc or /sys? I only see the max.

[SuperQ]

The other question is can this be obtained without root privileges.

[brian-brazil]

From my quick research we'd be talking walking all fds of all processes, which isn't going to work out.

[albertvaka]

Indeed, it looks like unfortunately this is not something that Linux exposes :/

I'm going to research a bit if this is something that could be patched in the kernel so it is exposed somewhere, maybe in /proc/sys/fs/inotify/.

[discordianfish]

fwiw, I think this would be very useful in general.

[zuzzas]

Is it feasible to create an inotify watch and increment a counter if it fails?

[albertvaka]

Via stackoverflow I've found this bash line that will tell you how many inotify watches are used per-process and per-user (assuming you run it as root, otherwise only the ones used by your user):

find /proc/*/fd -lname anon_inode:inotify 2> /dev/null |
   cut -d/ -f3 |
   xargs -I '{}' -- ps --no-headers -o '%p %U %c' -p '{}' |
   uniq -c |
   sort -nr

It traverses the entire /proc virtual filesystem and does a call to ps for each inotify node, but it seems fast enough for me. Is this something reasonable to do in node_exporter to get the metric?

[discordianfish]

@albertvaka That's not an option unfortunately since we don't allow features that require root to work. You can of course use that bash snippet and use the textfile collector.

[albertvaka]

In 2019 I sent a 3-commit patch to the Linux kernel to expose the number of inotify watches used and total in /proc:

[PATCH v3 1/3] Move *_ucounts functions above [PATCH v3 2/3] kernel/ucounts: expose count of inotify watches in use [PATCH v3 3/3] Documentation for /proc/sys/user/inotify

I got one response and I replied here. But then my answer got ignored and then I got a job and didn't have time to push this further. So it is not merged. Since I see this issue got some upvotes, maybe someone wants to pursue this further :)

[discordianfish]

@albertvaka Oh kudos for the effort. I don't think anyone on our side has time to follow up on that but if someone else steps up, that'd be great of course :)