watching over 1000+ files causing SIGABORT

[jayphelps]

The cause of this issue is very likely somewhere lower-level (libuv and/or OS X itself) but I'm documenting here because it's how I reproduced it and how I worked around it.

This is happening with node v0.11.13, grunt-contrib-watch v0.6.1

I've got a project that is watching over 1000 files and when a file changes, the spawned grunt task process is almost immediately aborted, regardless of the task it is asked to do.

After some trial and error I was able to keep it from crashing by changing stdio: "inherit" for the the grunt.util.spawn() to stdio: "pipe". Which means the output will be piped to our current processes stdio instead of actually sharing or "inheriting" ours.

I tried to find the exact file count that would cause the abort, but as far as I can tell it's not a set number. Always happened for counts above 1050, counts around 1010 would sometimes abort, sometimes not. My gut was 1024 for obvious reasons, but it did not seem to be the magic number. I still think this isn't a coincidence, though.

My ulimit/launchctl maxfiles limit has been indeed raised far above and no errors or warnings are provided by grunt nor does the spawned process return anything when it is killed other than the SIGABRT code. Particularly, the dreaded "Error: EMFILE, too many open files" is not logged anywhere I can find, which is normal error if you haven't bumped your limits.

Also of note, while grunt is watching, opening a new shell prompt and running the same attempted grunt task did not cause any aborts, so it doesn't appear to be permissions or locking related at first glance.

More info

Taking a look at the crash report, we can see the stack was aborted after the libuv OS X select call, which itself has several manual aborts that could have been thrown.

Thread 16 Crashed:
0   libsystem_kernel.dylib          0x00007fff8af84866 __pthread_kill + 10
1   libsystem_pthread.dylib         0x00007fff84e5e35c pthread_kill + 92
2   libsystem_c.dylib               0x00007fff86f7fb1a abort + 125
3   node                            0x000000010014d874 uv__stream_osx_select + 671
4   node                            0x0000000100146335 uv__thread_start + 25
5   libsystem_pthread.dylib         0x00007fff84e5d899 _pthread_body + 138
6   libsystem_pthread.dylib         0x00007fff84e5d72a _pthread_start + 137
7   libsystem_pthread.dylib         0x00007fff84e61fc9 thread_start + 13

[shama]

Could you try with the latest stable version of node.js? v0.10.x. Odd versions are considered unstable. That may or may not be the reason.

The specific number of files is likely not the cause (I've ran 1000+ files through it and did not get a SIGABRT). But rather a wall your machine is hitting at that number.

It would be interesting to find out why your machine is hitting that wall though. If you could make an example that reproduces this issue and post it for me to try and duplicate, that would be great. Thanks!

[jayphelps]

@shama Using v0.10.29, I get slammed with tons of

CarbonCore.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-21)

same as this report. I confirmed again that I am using v0.6.1.

I'll try and create a reproducible isolate project using node v0.11.13 (unless I can fix these FSEventStreamStart errors), but may be a while before I can spare the time since I'm not blocked by this anymore since I'm using the pipe workaround. I mostly wanted it documented in case someone else finds themself in the same boat.

[shama]

FWIW, the next version of gaze (the underlying library grunt-contrib-watch uses for watching files) will attempt it's own native solution. So far it seems to be able to watch a lot more files more reliably then node's core fs.watch but it's still in development. There are a couple of outstanding issues before we can upgrade it here but hopefully this will avoid these kinds of issues.

[timomayer]

any news here? got this problem now too cause my project grows and i have more and more files to watch