Verbose subprocess and c-library hangs with IPyKernel 6.x.x

[ElricleNecro]

Since IPyKernel 6.0.0, in a notebook that call some functions working with some very verbose subprocess and C/C++ library ends up hanging indefinitely.

This does not happens with IPyKernel 5.5.6, but can be circumvent, in 6.3.0 and above, by setting c.IPKernelApp.capture_fd_output = False in the config file.

From what I am guessing, this is a bug in the thread that handle the capture of stdout and stderrthat was introduced in 6.0.0.

While playing around with the configuration of the kernel and a notebook, it seems that the thread in IPyKernel which read stdout and stderr start reading one of them, but all output goes to the other at this moment. Since reading from stdout or stderr is a blocking operation, the read operation never stop and the other output ends up filling its buffer, at this stage, no one can write to the output, hence the kernel completely hangs.

For me this is probably it since, when we kill the kernel (from the notebook interface), a lot of output suddenly appeared, before having the stack trace for the KeyboardInterrupt exception. Just like if one of the output wasn't read for some time, but got flushed when interrupting the kernel.

[mdickinson]

#868 may be related