Directing output of out external threads/processes to specific cells

[krassowski]

https://github.com/ipython/ipykernel/pull/1186 made output of Python threads and asyncio tasks go to right cells. It would be interesting, although challenging to make it work for any thread/process (e.g. created from a C++ dependency).

Here are two possible options to consider:

• (a) define format (e.g. JSON-RPC) where each write to the stdio/stderr would be a packed with both target cell ID and a blob message (slight challenges related to buffering and format choice) - this is how LSP works.
• (b) allow to open file descriptors per-cell (*2 for stdout/stderr) and make their IDs easily accessible to the external processes for writing (this could use virtual filesystem on Linux), either by having an API return the file descriptor or by having them in predictable locations (e.g. relative to kernel runtime); of course the challenges here would be around OS-specific limits for watching files and the allowed number of open file descriptors.

Any thoughts on whether any of these would be in scope?

[krassowski]

@jasongrout (during server and kernels meeting): Sage used something like (b) but using folders rather than files; also a single cell can have outputs which are updated based on output ID rather than cell ID - something to consider.