"No space left on device" error coming from remote monitor

[offbyone]

I just upgraded to your 0.3.0 release, and I'm no longer able to have unison working. On the local side, I'm getting this error:

2022-03-11T19:06:28Z DEBUG unison_fsmonitor] event: Input("DONE\n")
Error: Io(Os { code: 28, kind: StorageFull, message: "No space left on device" })
Fatal error: Server: File monitoring helper program not running
[2022-03-11T19:07:06Z DEBUG unison_fsmonitor] event: Input("")
[2022-03-11T19:07:06Z DEBUG unison_fsmonitor] >> ERROR Unrecognized%20cmd%3A%20

I believe that this is being emitted by the unison-fsmonitor on my remote host, which is being built from the current tip of your master branch.

I am not, as far as I can tell, out of space, either locally or remotely.

$ df -i
Filesystem       Inodes    IUsed    IFree IUse% Mounted on
devtmpfs       24200302      542 24199760    1% /dev
tmpfs          24202461        2 24202459    1% /dev/shm
tmpfs          24202461      707 24201754    1% /run
tmpfs          24202461       16 24202445    1% /sys/fs/cgroup
/dev/nvme0n1p1 31457280 17799544 13657736   57% /
tmpfs          24202461        5 24202456    1% /run/user/272766
tmpfs          24202461        1 24202460    1% /run/user/0
tmpfs          24202461        1 24202460    1% /run/user/221009

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         93G     0   93G   0% /dev
tmpfs            93G     0   93G   0% /dev/shm
tmpfs            93G  2.1M   93G   1% /run
tmpfs            93G     0   93G   0% /sys/fs/cgroup
/dev/nvme0n1p1  473G  312G  161G  66% /
tmpfs            19G     0   19G   0% /run/user/272766
tmpfs            19G     0   19G   0% /run/user/0
tmpfs            19G     0   19G   0% /run/user/221009

(all of this is in a directory on the / mount)

I've "worked around" this by switching to a different fsmonitor, which is ... okay, I guess, but yours has been the most consistently stable of the ones I've used so far, and I would prefer to keep using it if I could. I'm not sure how to debug this on the server side, though.

[offbyone]

Interestingly, this only happens if I watch a very large tree of files. If I watch a subset of it, the sync is fine, but if I watch the large tree it's not.

As far as I can tell, my inotify limits are higher than the number of files in play.

[autozimu]

Interesting. I've been using this tool to monitor directories of several gigabytes (sources + compiled artifacts) and I haven't see this error.

The only plausible reason might be some watching limit is hit, like https://github.com/pantsbuild/pants/issues/10408#issuecomment-661631257

Either way, it's tricky to trouble shoot without being able to reproduction and it's unlikely an issue of the tool itself.