WorkerCluster occasionally crashing.

[happilymarrieddad]

Hey guys,

Very rarely (maybe once a month) I'll get this on one of our containers and it crashes that server. I have to restart the server to get it to respond. I'm not sure if it's a socketcluster problem but non of our other services experience this. We have 4 application servers in our SCC cluster. It doesn't affect users really because the rest of the cluster takes care of the load. Thanks!

1503450314685 - Origin: WorkerCluster (PID 4916)
   [Error] Error: spawn /usr/bin/nodejs ENOENT
    at _errnoException (util.js:1022:11)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:192:19)
    at onErrorNT (internal/child_process.js:374:16)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
1503450314685 - Worker 0 exited - Exit code: 1
1503450314685 - Origin: WorkerCluster (PID 4916)
   [Error] Error: spawn /usr/bin/nodejs ENOENT
    at _errnoException (util.js:1022:11)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:192:19)
    at onErrorNT (internal/child_process.js:374:16)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
1503450314957 - Worker 6 exited - Exit code: 1
1503450314957 - Origin: WorkerCluster (PID 4916)
   [Error] Error: spawn /usr/bin/nodejs ENOENT
    at _errnoException (util.js:1022:11)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:192:19)
    at onErrorNT (internal/child_process.js:374:16)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
1503450314957 - Worker 5 exited - Exit code: 1
1503450314957 - Origin: WorkerCluster (PID 4916)
   [Error] Error: spawn /usr/bin/nodejs ENOENT
    at _errnoException (util.js:1022:11)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:192:19)
    at onErrorNT (internal/child_process.js:374:16)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
1503450314957 - Worker 3 exited - Exit code: 1
1503450314958 - Origin: WorkerCluster (PID 4916)
   [Error] Error: spawn /usr/bin/nodejs ENOENT
    at _errnoException (util.js:1022:11)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:192:19)
    at onErrorNT (internal/child_process.js:374:16)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
1503450314958 - Worker 2 exited - Exit code: 1
1503450314958 - Origin: WorkerCluster (PID 4916)
   [Error] Error: spawn /usr/bin/nodejs ENOENT
    at _errnoException (util.js:1022:11)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:192:19)
    at onErrorNT (internal/child_process.js:374:16)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
1503450314958 - Worker 1 exited - Exit code: 1
1503450314958 - Origin: WorkerCluster (PID 4916)
   [Error] Error: spawn /usr/bin/nodejs ENOENT
    at _errnoException (util.js:1022:11)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:192:19)
    at onErrorNT (internal/child_process.js:374:16)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)
1503450314958 - Worker 7 exited - Exit code: 1
1503450314958 - Origin: WorkerCluster (PID 4916)
   [Error] Error: spawn /usr/bin/nodejs ENOENT
    at _errnoException (util.js:1022:11)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:192:19)
    at onErrorNT (internal/child_process.js:374:16)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)

[jondubois]

@happilymarrieddad When you get this error, does /usr/bin/nodejs still exist? If it's inside a container then you can try to SSH into the container to check if that path exists.

What happens if you run /usr/bin/nodejs -v in the command line?

[happilymarrieddad]

v8.11.2

It's on an EC2 instance. It's not running inside a container like docker. It's weird and really hard to diagnosis because it's so rare. Next time it happens, I'll check to see if /usr/bin/nodejs still exists. I had to restart the server so I don't know. Thanks!

[mahmoudsalem]

[Busy] Launching SocketCluster node[23351]: pthread_create: Resource temporarily unavailable

Broker PID: 23351 WorkerCluster PID: undefined 1543746594818 - Origin: WorkerCluster [Error] Error: spawn /usr/bin/node EAGAIN at _errnoException (util.js:1024:11) at Process.ChildProcess._handle.onexit (internal/child_process.js:190:19) at onErrorNT (internal/child_process.js:372:16) at _combinedTickCallback (internal/process/next_tick.js:138:11) at process._tickCallback (internal/process/next_tick.js:180:9)

i also get this error @jondubois

[jondubois]

@mahmoudsalem That looks more like a Node.js/OS issue than specifically an SC issue. Maybe your system is out of resources? Reached file/socket limit, process limit, memory limit or similar.