python-driver icon indicating copy to clipboard operation
python-driver copied to clipboard

Stop fallback to asyncio on python3.12

Open fruch opened this issue 11 months ago • 5 comments

during adding the support to python3.12 we took a call to fallback to asyncio once asyncore isn't available.

seems like doing so, we are running into issues that we didn't had coverage for, like using asyncio event loop, from actual async code. since we don't have such in the integration suite of this driver, nor in dtest.

it was hit only by scylla core test.py, that is build as async code

so we should do one of the two, or even both:

  • adapt the same logic as upstream to fail when we can find any event loop available, and not default to asyncio
  • add some coverage for the async code usages, to cover the issue we were hitting, and fix them

fruch avatar Mar 14 '24 08:03 fruch

Who should do this one?

roydahan avatar Mar 18 '24 12:03 roydahan

Who should do this one?

Someone, if we wouldn't core developers are gonna hit by that once moving to python3.12

fruch avatar Mar 18 '24 16:03 fruch

@fruch In the readme there is a statement that implies that this driver should not (yet) be used with python 3.12

https://github.com/scylladb/python-driver/blob/master/README.rst?plain=1#L13

Is this still true, or did you already implement support for python 3.12?

I am asking this, because I have a project based on python 3.12, asyncio, uvloop (but I could remove uvloop if needed), OSS Scylla 5, and I am considering to use your driver.

aisven avatar Jul 21 '24 20:07 aisven

@fruch In the readme there is a statement that implies that this driver should not (yet) be used with python 3.12

https://github.com/scylladb/python-driver/blob/master/README.rst?plain=1#L13

Is this still true, or did you already implement support for python 3.12?

we do support python 3.12

I am asking this, because I have a project based on python 3.12, asyncio, uvloop (but I could remove uvloop if needed), OSS Scylla 5, and I am considering to use your driver.

if wouldn't recommend using the asyncio eventloop, it's still experimental, and some parts of it isn't working, for example SSL.

you can still write you code using asyncio, all of the other eventloops would work regardless.

fruch avatar Jul 21 '24 21:07 fruch

@fruch Thank you for the reply. I like asyncio as my service is completely based on it, so I do not switch for now.

The service I am developing is streaming a lot of time series data from Scylla as a data source to some other data sink (think Kafka or TimescaleDB or whatever). Also, in one service instance I can configure to have not one, but multiple such streams running concurrently, i.e. one such stream per keyspace (there is one relevant table in each keyspace). Thus it is an interesting scenario with significant load.

The contents in each keyspace (relevant table) are large, but still in a ballpark that allows me to use just one stream per keyspace, i.e. I do not need to shard the streaming of the data regarding any particular keyspace.

I am making use of the normal good old paging facility of the driver API, so my design is effectively a type of micro-batched streaming.

Regrading the driver API I use:

  1. session.execute_async to execute a simple SELECT * query which does a paged full table scan
  2. future.result to eventually get the current rows of the current page
  3. result.fetch_next_page to forcibly and explicitly fetch the next page

I also employ facilities from asyncio to run synchronously waiting stuff on an executor and to properly await futures, respectively:

  • loop.run_in_executor to dispatch 2 and 3
  • asyncio.wait_for to properly await futures

aisven avatar Jul 27 '24 15:07 aisven