Async support in Python

[pskopnik]

Async support for the splitstream Python library would be much appreciated. Python I/O applications are increasingly moving to rely on the async/await semantics and so far there is no way of using splitstream for reading JSON objects in these cases.

Specifically, async support for me means a new variant of splitfile, which:

• accepts filelike.read(n) returning an awaitable object. This would match both the asyncio.StreamReader interface and the aiofiles.(File) interface, covering most async use cases for splitstream.
• either:
  • exposes an async iterator instead of a normal iterator.
  • callback() may return an awaitable object which will be awaited before continuing to read.

There are two obvious ways of providing async support:

• Expose the basic C interface of splistream to Python. Via SplitstreamGetNextDocument, callers can pass in their own data. Async variants of splistream can then be implemented in Python on top of this interface. An async splitfile() should probably be implemented in this project, so that there is a reliable reference implementation for the most common case.
• Implement an async_splitfile() function (name to be decided) directly in C by implementing the async iterator protocol. Possibly, the existing splitfile could attempt to detect whether the caller intends sync or async support. But providing a separate function seems clearer in any case.

To me, the second approach sounds more interesting and would match exactly what the current splitstream library does. It could also reuse most of the existing code (after some refactoring) boiling down to a more complex state machine for the generator.

I'm available to provide an initial implementation. Although I've not yet worked with the async protocols in C extensions, I've looked at the underlying concepts and am confident to get something working (open questions: how to implement send()/throw()/close() for the iterator representing the awaitable :thinking: ).

[rickardp]

I agree that the second approach would be more appropriate. Note that to be performant, file I/o is offloaded to the C code rather than calling the reader to get the bytes. A lot of string copying is avoided that way, this makes a huge difference. I wonder if this also means async file I/o would be needed for this to be useful in practice? As as it is usually not for file I/o you’re looking at async, I suspect what would really be needed is sockets.

Do you have a concrete use case for this? Maybe it would help starting to look from that end?