async subprocess with live output (on Windows requires ProactorEventLoop which ipykernel doesn't use)

[flutefreak7]

I'd like to be able to run subprocesses in the notebook and trace their output using asyncio. I thought that since the ipykernel is now using asyncio that it should just be a fairly simple await call.

https://blog.jupyter.org/ipython-7-0-async-repl-a35ce050f7f7

However, when I try to run the following example:

from https://fredrikaverpil.github.io/2017/06/20/async-and-await-with-subprocesses/

async def run_command(*args):
    """Run command in subprocess
    
    Example from:
        http://asyncio.readthedocs.io/en/latest/subprocess.html
    """
    # Create subprocess
    process = await asyncio.create_subprocess_exec(
        *args,
        # stdout must a pipe to be accessible as process.stdout
        stdout=asyncio.subprocess.PIPE)

    # Status
    print('Started:', args, '(pid = ' + str(process.pid) + ')')

    # Wait for the subprocess to finish
    stdout, stderr = await process.communicate()

    # Progress
    if process.returncode == 0:
        print('Done:', args, '(pid = ' + str(process.pid) + ')')
    else:
        print('Failed:', args, '(pid = ' + str(process.pid) + ')')

    # Result
    result = stdout.decode().strip()

    # Return stdout
    return result

I get the following:

---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
cell_name in async-def-wrapper()

<ipython-input-30-b0d1433a59c3> in run_command(*args, **kwargs)
      6     """
      7     # Create subprocess
----> 8     process = await asyncio.create_subprocess_exec(*args)
      9 
     10     # Status

c:\python37\lib\asyncio\subprocess.py in create_subprocess_exec(program, stdin, stdout, stderr, loop, limit, *args, **kwds)
    215         program, *args,
    216         stdin=stdin, stdout=stdout,
--> 217         stderr=stderr, **kwds)
    218     return Process(transport, protocol, loop)

c:\python37\lib\asyncio\base_events.py in subprocess_exec(self, protocol_factory, program, stdin, stdout, stderr, universal_newlines, shell, bufsize, *args, **kwargs)
   1531         transport = await self._make_subprocess_transport(
   1532             protocol, popen_args, False, stdin, stdout, stderr,
-> 1533             bufsize, **kwargs)
   1534         if self._debug and debug_log is not None:
   1535             logger.info('%s: %r', debug_log, transport)

c:\python37\lib\asyncio\base_events.py in _make_subprocess_transport(self, protocol, args, shell, stdin, stdout, stderr, bufsize, extra, **kwargs)
    461                                          extra=None, **kwargs):
    462         """Create subprocess transport."""
--> 463         raise NotImplementedError
    464 
    465     def _write_to_self(self):

NotImplementedError: 

My initial research shows that on Windows I have to use the Proactor event loop to do subprocesses? So does this mean that ipykernel simple doesn't use the event loop that I would need to do this? Or am I missing something?

[flutefreak7]

What's ironic to me is that I can already do this using %gui qt and QProcess. I'm building notebook widgets to replace / complement some of my PyQt stuff and I thought that have a batch task monitor dashboard in the notebook would work better with asyncio so that I wouldn't require a Qt event loop to update progress bars.

• Perhaps using a GUI event loop isn't a bad idea for what I'm doing
• Perhaps I should investigate using threads?
• Maybe there's a library that watches Windows processes and turns their pipes effectively into sockets so that the Selector event loop can work better with them? I'm out of my depth...

[raythurnvoid]

I managed to overcome this issue with just Popen and asyncio.to_thread: https://stackoverflow.com/a/76981596/1951947