Machinery for translating pipe stream into messages

[takluyver]

This is the initial piece for capturing output from the kernel at the OS level, by hooking stdout and stderr up to pipes.

Kernels wanting to make use of this machinery will need to delimit the stream output resulting from executing a cell with markers containing the execute message ID:

b"""
\0JUPYTER STREAM BEGIN d853e19e-012f-438b-ac71-6970f49a53b7
bar
baz
\0JUPYTER STREAM END d853e19e-012f-438b-ac71-6970f49a53b7
"""

These are used to set the message id in the parent header of the messages generated, so that a frontend can associate them with the relevant cell. The end marker is translated into an EndOfOutput object - once the frontend has seen one of these from each captured stream, plus the kernel-idle message on iopub, it knows it has all output produced while that cell was executing. The cell may have started background threads, and output from those may be lost or misdirected, but that's already the case, and I think it's unavoidable for piped data. I believe this machinery should recover if e.g. one of the markers it looks for isn't there.

Question: at present, output outside the marked blocks (after one END, before the next BEGIN) is silently discarded. Do we want to expose it to the frontend in some way, e.g. as stream output messages without a parent header?

[minrk]

Do we want to expose it to the frontend in some way, e.g. as stream output messages without a parent header?

I think so, or preserve the current behavior of associating it with the most recently seen parent.