Support stdout and stderr on the worker

[logikaljay]

I want to stream the stdout and stderr of the tasks to a log file/database/websocket etc.

I think it is achievable, providing a change is made to the options provided to the Worker constructor here: https://github.com/piscinajs/piscina/blob/2f9567883fe783036c570acd7e0c1385cec1941f/src/index.ts#L570-L577

I implemented this locally and it seemed to work.. mostly.. (happy to open a PR). The issue I found was that the task never ended.. I couldn't figure this out, it may have been my stream.. I used fs.createWriteStream to see if it would create a log file of the tasks stdout, which it did. However as I described, when the task finished, the application did not exit as it does when stdout is set to false on the Worker.

There is probably something I am missing, especially when it comes to the stdout and stderr checks being made here: https://github.com/piscinajs/piscina/blob/2f9567883fe783036c570acd7e0c1385cec1941f/src/worker.ts#L158-L167

There were also some warnings in the console about a "Possible EventEmitter memory leak" being detected:

(node:3721611) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 17 error listeners added to [WriteStream]. Use emitter.setMaxListeners() to increase limit

I believe this is because I am adding the same listener to each worker. setting setMaxListeners higher than the number of CPU cores resolved this warning - this is probably not the correct way to go about this though.

Any thoughts/comments would be appreciated. Hopefully I am not wasting anyones time.

[jasnell]

@addaleax would likely be able to explain what's happening here a lot better than I can since she wrote that part of the workers code... generally tho... process.stdout and process.stderr in workers essentially forward the data on to main thread for processing. What would appear to be happening here is that the write callbacks might not be getting called.

[taxilian]

my guess would be that if it's not closing the thread it has something to do with needing to close at least stdin, maybe both, when it's time to shut down the thread.

[metcoder95]

Have you tried setting useAtomics: false?