python-driver
python-driver copied to clipboard
Stop fallback to asyncio on python3.12
during adding the support to python3.12 we took a call to fallback to asyncio once asyncore isn't available.
seems like doing so, we are running into issues that we didn't had coverage for, like using asyncio event loop, from actual async code. since we don't have such in the integration suite of this driver, nor in dtest.
it was hit only by scylla core test.py, that is build as async code
so we should do one of the two, or even both:
- adapt the same logic as upstream to fail when we can find any event loop available, and not default to asyncio
- add some coverage for the async code usages, to cover the issue we were hitting, and fix them
Who should do this one?
Who should do this one?
Someone, if we wouldn't core developers are gonna hit by that once moving to python3.12
@fruch In the readme there is a statement that implies that this driver should not (yet) be used with python 3.12
https://github.com/scylladb/python-driver/blob/master/README.rst?plain=1#L13
Is this still true, or did you already implement support for python 3.12?
I am asking this, because I have a project based on python 3.12, asyncio, uvloop (but I could remove uvloop if needed), OSS Scylla 5, and I am considering to use your driver.
@fruch In the readme there is a statement that implies that this driver should not (yet) be used with python 3.12
https://github.com/scylladb/python-driver/blob/master/README.rst?plain=1#L13
Is this still true, or did you already implement support for python 3.12?
we do support python 3.12
I am asking this, because I have a project based on python 3.12, asyncio, uvloop (but I could remove uvloop if needed), OSS Scylla 5, and I am considering to use your driver.
if wouldn't recommend using the asyncio eventloop, it's still experimental, and some parts of it isn't working, for example SSL.
you can still write you code using asyncio, all of the other eventloops would work regardless.
@fruch Thank you for the reply. I like asyncio as my service is completely based on it, so I do not switch for now.
The service I am developing is streaming a lot of time series data from Scylla as a data source to some other data sink (think Kafka or TimescaleDB or whatever). Also, in one service instance I can configure to have not one, but multiple such streams running concurrently, i.e. one such stream per keyspace (there is one relevant table in each keyspace). Thus it is an interesting scenario with significant load.
The contents in each keyspace (relevant table) are large, but still in a ballpark that allows me to use just one stream per keyspace, i.e. I do not need to shard the streaming of the data regarding any particular keyspace.
I am making use of the normal good old paging facility of the driver API, so my design is effectively a type of micro-batched streaming.
Regrading the driver API I use:
-
session.execute_async
to execute a simple SELECT * query which does a paged full table scan -
future.result
to eventually get the current rows of the current page -
result.fetch_next_page
to forcibly and explicitly fetch the next page
I also employ facilities from asyncio to run synchronously waiting stuff on an executor and to properly await futures, respectively:
-
loop.run_in_executor
to dispatch 2 and 3 -
asyncio.wait_for
to properly await futures