botocore

botocore copied to clipboard

This is a tracking issue for the feature request of supporting asyncio in botocore, originally asked about here: https://github.com/boto/botocore/issues/452

There's no definitive timeline on this feature, but feel free to +1 (thumbs up 👍) this issue if this is something you'd like to see. Also, if you have any additional information about asyncio you'd like to share (even just about your specific use case) feel free to chime in.

Thanks for logging this! Currently I'm only calling list_distributions(..) and create_distribution(..) on a boto3.client('cloudfront') instance, which I know is a small fraction of the boto3 API, but it'd be nice if those calls cooperatively yielded in between making the request and receiving the response so that other coroutines in my app could run in the meantime. (I could always just roll my own client using aiohttp too since I'm using so little of the API, but figured it was worth asking about anyway.)

In case it helps, aiodns provides an example of supporting asyncio on Python 2.6+ (see https://github.com/saghul/aiodns#python-versions).

If boto3 was already written with the assumption of blocking rather than asynchronous I/O throughout, I'm not sure how disruptive a change this would be.

I'm not that familiar with the internals of botocore, but I think the right approach is to create a custom version of endpoint.py and client.py that used aiohttp under the hood and yielded as you want.

I would probably start by subclassing Endpoint and making an aiohttp version of _send_request and _get_request. I'd also subclass ClientCreator and override _create_api_method (that is where the calls to the endpoint come from). I haven't yet figured out how i'd get from a service object to a client created with my ClientCreator subclass.

Does that sound right, @jamesls? I might have a go...

I've started a port of botocore to asyncio, at:

https://github.com/rdbhost/botocore

So far, the S3 integration tests pass, as do about 2/3 of the unit tests. I hope to have all tests converted and passing by a week from now.

The port depends on yieldfrom.requests and yieldfrom.urllib3 , asyncio ports of the requests and urllib3 libraries.

@rdbhost Awesome! Thanks for working on this. Definitely interested to follow your progress. @jamesls / botocore maintainers, had a chance to check this out / evaluate for merge potential?

+1

+1

This issue has gotten a couple of +1s in the last few days, so I thought I would point out that the work is done, at:

https://github.com/rdbhost/yieldfromBotocore

It implements an asyncio version of botocore, not boto3. I may eventually convert boto3, but my own needs have been satisfied by botocore, so other things now have priority.

David

On Thu, Jun 18, 2015 at 6:59 AM, AlexNigl [email protected] wrote:

+1

— Reply to this email directly or view it on GitHub https://github.com/boto/botocore/issues/458#issuecomment-113148024.

+1

@rdbhost, thanks for tackling that! It seems, however, that https://github.com/rdbhost/yieldfromBotocore is 271 commits behind upstream. Are you actively maintaining it?

My intention has been to do a merge when upstream reached v1.0; that seems to have happened without my noticing.

I will be merging commits from upstream, this weekend.

+1 @jamesls are you planning on pulling rdbhost's changes into botocore itself? We are very interested in this feature.

It is not a very promising candidate for merging back into botocore.

The changes are numerous, and the changes needed to make botocore functional within asyncio make it non-functional outside asyncio. I expect it to be a seperate product indefinitely, with a parallel API, meaning an API as similar as possible within the asyncio constraints.

One thing that also complicates things is that botocore supports as far back as python 2.6.5, so we'll need to figure out how we can support asyncio and still maintain py2 support. I see that there are asyncio backports to python2, so perhaps something could be done there.

Trollius (asyncio port to Python 2.x) is complete and stable.

+1

I have working port of botocore for asyncio: https://github.com/jettify/aiobotocore using aiohttp for async http requests. I am trying to reuse as much botocore code as possible, so I patched only several classes and just import rest of the code as result library has few hundreds lines of code. And this approach helps to keep up with upstream, but obvious downside, I rely on internal interfaces which is subject of change for new libs. API almost the same as botocore just yield from or awaite (python 3.5) should be added before calls.

For now I am using aiobotocore with s3 and ported almost all s3 test, except I need to work more on pagination since it is not easy to implement iterator protocol with yield from.

+1 integrating aiobotocore might be doable

+1

+1

+1

+1

OK guys, I've written a production ready, and hopefully useful to the community, asyncio extension to botocore. The library is currently included in https://github.com/quantmind/pulsar-cloud in the asyncbotocore module but it is self-contained and could be striped out if needed.

The API is the same as botocore but with the addition of http_session keyword which is an optional asyncio compatible HTTP client (it must expose the _loop attribute) and API similar to requests. As far as I know there are two of such clients:

• pulsar http client
• aiohttp http client (@asvetlov does aihttp client/session have the _loop attribute?)

If used with pulsar, the library can also use greenlet in an implicit asynchronous fashion. Check https://github.com/quantmind/pulsar-cloud for more info

By the way, thanks to @jettify for the initial effort from which I leveraged from

Feedbacks welcome!

@lsbardel yes, aiohttp.ClientSession has _loop but I don't encourage using private attributes. BTW we use https://github.com/jettify/aiobotocore as aiohttp-based library.

@asvetlov cool, I understand the private attribute thing, but the _loop attribute should be (almost) a standard for asyncio objects ;-) maybe we should ask guido.

I would be happy to use https://github.com/jettify/aiobotocore has low level async botocore library but currently it does not work with pulsar http client. At the moment it clearly requires aiohttp while cloud.asyncbototore does not require an asyncio library as such.

So maybe we should converge to a library that allows for different http clients?

By the way all that is python 3.4 or above

@lsbardel At asyncio active development times Guido was against public loop attribute, he motivated it as "user always may pass explicit loop if needed". I don't think the decision has changed now.

+1

+1

+1

+1