[Tracking] roll into botocore

[thehesiod]

just wondering, thanks!

[jettify]

I do not object, just do not see clean way how to accomplish this. Could be tricky because botocore also supports python2.7 and aiobotocore uses yield from heavily.

[thehesiod]

perhaps making the aio portion available for python3.x only? I think they do a similar thing with pyzmq: http://pyzmq.readthedocs.org/en/latest/api/zmq.asyncio.html

[thehesiod]

I see this is being tracked in https://github.com/boto/botocore/issues/458

[jettify]

the best way to handle this issue if client could be somehow abstracted from query/url/headers builder from IO. Nice approach also used here https://github.com/cablehead/python-consul

[graingert]

@jettify an option for abstraction of IO from any code: https://gist.github.com/graingert/ca6cdd5d9ae2e18ca917b4594ac8a633

[jettify]

We need to start thinking how to patch botocore so we can plugging our code using public api, for instance first easy change is to update https://github.com/boto/botocore/blob/develop/botocore/session.py#L108-L109 Session object to accept ClientCreator, so in aiobotocore we just pass AioClientCreator without coping screen of code.

[thehesiod]

perhaps we can have a class by class goal of rolling into botocore to make it bite-sized chunks of work. But I think before that we need a high-level strategy defined of how we want this to integrate into botocore to create a straw-man we can work towards and get botocore dev approval on the design.

[thehesiod]

another option is switching to the c++ library: https://github.com/aws/aws-sdk-cpp this would greatly increase performance as well, mainly due to signing.

[dazza-codes]

The general pattern in botocore is to register components or to hook up callbacks to an event-emitter (which itself could be a component). See

• https://boto3.amazonaws.com/v1/documentation/api/latest/guide/events.html#boto3-specific-events
• the boto3 example shows how to replace the BaseClient with a substitute parent class, which could be AioBaseClient
• it also shows how to add custom methods on a client; aiobotocore could add new async methods that could be used side-by-side with existing "synchronous" methods rather than replacing those methods (if the botocore API is preserved and extended with new async methods, it might have a better chance of inclusion in botocore)
• the boto3 event documentation is better than botocore event documentation, but essentially the same event system is used in botocore and boto3
• moto uses the event system to apply mocks as before-send callbacks

In that context, aiobotocore might create and register components for:

• custom event emitter that uses coroutines that can be awaited
• custom client creator and client that can await event callbacks
• custom client event handlers/coroutines that can be awaited

It’s not clear whether this would be any different, in the end, than what aiobotocore is already. This might be an alternative pattern to explore in aiobotocore as a POC, but it’s not clear whether it would work and it’s got little chance, if any, of ever getting merged into botocore (lots of useful PRs are stale on botocore).