rethinkdb-python
rethinkdb-python copied to clipboard
rethinkdb
Adding `async` and `await` support for Python 3.5
Hi guys!
With the advent of async and await keywords in Python 3.5, I think we need to add some convenience methods to the connection object that is returned when we make a r.connect(... function call. This would not be difficult to do, because we can already do a yield r.connect anyways.
So, the proposal is to add an __await__ method to the connection object. This will not be hard to implement though, as you can see with aiohttp's await implementation:
@asyncio.coroutine
def __iter__(self):
resp = yield from self._coro
return resp
if PY_35:
def __await__(self):
resp = yield from self._coro
return resp
@gamesbrainiac - I'm not entirely sure what this issue is proposing. I tried out the existing asyncio implementation of the Python driver, and it appears to work correctly with async and await:
import rethinkdb as r
import asyncio
r.set_loop_type('asyncio')
async def run():
conn = await r.connect('localhost')
print('res: %d' % await r.expr(1).run(conn))
await conn.close()
asyncio.get_event_loop().run_until_complete(run())
There is the possibility of extending the AsyncioCursor object to support __aiter__ according to PEP 0492. This would be fairly useful, as you currently cannot use cursors as iterators in any of the async backends for the Python driver.
How does this work when neither Connection or ConnectionBase have a __await__ method inside of net_asyncio?
And yes, the idea was to implement both asynchronous iteration (__aiter__) as well as asynchronous context managers (__aenter__ and __aexit__) .
However, since doing nothing to the existing class worked, I'm curious to see of seeing if setting __await__ = __iter__ does the trick.
Subscribing to this.
And yes, the idea was to implement both asynchronous iteration (aiter) as well as asynchronous context managers (aenter and aexit) .
That's really needed right now. I wonder how that'll behave with python3.4's decorated-style coroutines
However, since doing nothing to the existing class worked, I'm curious to see of seeing if setting await = iter does the trick.
I'd avoid this kind of things, they're obscure and make bugs hard to debug.
AFAIRemember @gamesbrainiac my PR I linked below the await was working in the use case you described, but we held it back because the async for ... in ... and/or async with .... syntaxes were not complete and our Python tests were limited in testing the results properly. But you're welcome to try it out or give us a hand if this is your area of expertise!
Related to rethinkdb/rethinkdb#5354
Really need this working, but it's not clear what the state of this feature is. rethinkdb/rethinkdb#5354 is waiting on a release according to @danielmewes. Are you accepting bug bounties to get this fixed sooner? If so, I can throw some cash at it. Or assistance with testing etc. Whatever is needed
Nudge.
I believe it's been about 4 months since I asked in the RethinkDB Slack channel. I was told, "that will be supported soon", and was sent here.
What's it going to take to get this feature added? Despite RethinkDB being a commercial company, I'm willing to donate $50 personally. @foxx mentioned a bug bounty. I'm sure there are others willing to pay to expedite this. I would offer a patch, but that's a paradox, because this feature is the thing holding me back from my first foray into RethinkDB (i.e., I know little about the source, etc.).
I'm moving this into the 2.3.x milestone, so that we can release it as a minor update before RethinkDB 2.4.
@foxx @orokusaki would you guys mind testing out this branch here of the Python driver if you still have some time to give it a look?
https://github.com/rethinkdb/rethinkdb/pull/5354
Awesome, thanks!
I just got into Europe for vacation (left my notebook home), but I will be testing this the moment I get back (toward end of month).
Sent from my iPhone
On Jul 9, 2016, at 12:24 AM, Daniel Alan Miller [email protected] wrote:
@foxx @orokusaki would you guys mind testing out this branch here of the Python driver if you still have some time to give it a look?
rethinkdb/rethinkdb#5354
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.
@foxx @orokusaki Have you had a chance to try out the driver from branch dalanmiller_aiter_and_anext in the meantime?
Hey @danielmewes sorry I haven't had chance to jump on this yet, I've had to jump on a different internal project for a few weeks but should have chance to circle round and try this again very soon!
Any progress on the pr/issue? https://github.com/rethinkdb/rethinkdb/pull/5354
I can see this has been added in: https://github.com/rethinkdb/rethinkdb/commit/dda2fea6326182350c7e845378efc6749c49ec13
When will a new driver release with this come out? I want to use this, and cannot seem to manage to get the driver installed from the github itself
Hey @Phxntxm, I can't answer the "when does it come out" question, but I can explain how to install from GitHub. (There may be an easier way, but this is what worked for me.)
- Get a copy of the repo (e.g. clone or download archive).
- Go into to the rethinkdb root directory.
- (I use Docker here but you could skip this step)
docker run --rm -v $PWD:/rethinkdb -it ubuntu:16.10 /bin/bash - (Now inside the docker container, you can skip this step if not using docker)
cd /rethinkdb apt update- Install build deps:
apt install build-essential curl libprotobuf-dev libcurl4-openssl-dev libboost-all-dev libncurses5-dev libjemalloc-dev protobuf-compiler python3.6 wget m4 - Install pip:
curl https://bootstrap.pypa.io/get-pip.py > /tmp/get-pip.py && python3.6 /tmp/get-pip.py && rm /tmp/get-pip.py - Set version string:
echo "version = '2.3.6.next'" > drivers/python/rethinkdb/version.py - Configure build:
./configure --allow-fetch python=/usr/bin/python3.6 - Make Python egg:
make py-sdist - Do a test install of egg:
pip3.6 install /rethinkdb/build/packages/python/rethinkdb-2.3.6.next.tar.gz - If that works, move the egg to wherever you need to deploy and
pip3.6 installit.
@dalanmiller @danielmewes Great progress on this! I have been using a build off of next branch in a project I'm working on. One thing I'm missing is an async context manager for AsyncioCursor. Is that in the works? I quickly wrote up this little kludge for the time being:
class CursorContext:
''' A context manager for an AsyncioCursor. '''
def __init__(self, query, conn):
''' Constructor. '''
self._query = query
self._conn = conn
self._cursor = None
async def __aenter__(self):
''' Context manager interface. '''
self._cursor = await self._query.run(self._conn)
return self._cursor
async def __aexit__(self, exc_type, exc, tb):
''' Context manager interface. '''
await self._cursor.close()
And its used like this:
async with self._db_pool.connection() as conn:
async with CursorContext(extract_query, conn) as cursor:
async for extract_doc in cursor:
pass
The context manager is really helpful for handling exceptions correctly, particularly notable in asyncio because cancelling a task that's currently looping over a cursor raises CancelledError. The implementation would be more concise if it was implemented inside AsyncioCursor itself.
The downside is that AsyncioCursor is also a synchronous context manager (has __enter__ and __exit__ methods inherited from Cursor), which is wrong because the synchronous implementation would never work for an async cursor. Those methods should probably be moved into DefaultCursor or to some new class (BaseSyncCursor?) so they don't get inherited by AsyncioCursor.
I'm happy to supply a PR but would need to have these questions answered before doing so.
