[Experimental] add asyncio support

[garrettheel]

I will likely break this up into separate PRs if we decide to move ahead, but opening this up as a place to discuss what asyncio support might look like.

The approach is inspired by this comment in python-trio, which details an interesting way to run async functions in a sync context, so long as they're actually synchronous under the hood. This allows us to write a single async implementation, and then dynamically generate the sync interface.

As far as the public interface goes, this currently leaves the sync interface unchanged and exposes _async variants for methods. I've only implemented the base connection so far, so this is far from complete.

I'm interested in any feedback on the approach and the interface in particular.

[svix-yair]

Hey @garrettheel, Can you please shed some light on why you abandoned this PR? We are experimenting with Pynamodb and we need async support.

It would be great to give us some pointers and share any information you already have so we can help adding the Async support for Pynamodb.

[garrettheel]

hey @svix-yair, apologies for forgetting to leave an update here. I'll need to page this back in, but I think the main issue that I hit was aiobotocore being the only library that supports this and:

1. it brings along a large number of dependencies, where we've typically tried to keep deps very minimal for PynamoDB
2. it's not sanctioned by AWS or officially supported. there's little to suggest that this will stick around for the long term

I was waiting/hoping that botocore would implement asyncio natively, which would be a much better path to supporting it, but that still hasn't happened sadly. The other alternative is to forgo botocore for everything but a few utility tasks (e.g request signing, auth) and construct/send the requests ourselves (either using lower-level native python APIs or with something like httpx).

I'm definitely still interested in making progress on this and would welcome any help from others.

[tasn]

Hey @garrettheel, thanks for all of the information and all of your work on pynamo! I work with @svix-yair, and I've been trying to get an async pynamo working for us.

I understand your concerns about using a 3rd party not sanctioned by AWS, though realistically AWS isn't going to have an async version any time soon, so a community effort is the best we've got. I think the aiobotocore people aren't going anyway, it's the same ones behind apg and aiohttp (which aiobotocore uses). Additionally, many of the deps are just other deps of their own projects. In summary, I don't think it's a massive concern.

The httpx idea sounds very promising, though it also sounds like a lot of work and essentially re-implementing aiobotocore. I think aiobotocore just uses aiohttp so maybe you'd have better luck replacing aiohttp there with something that doesn't pull as many deps?

I wish I could have helped you with all of the above, though tbh, it sounds like a very difficult task for an AWS noob like myself, especially since this task requires deep AWS expertise, or at least deep understanding of how to interact with the AWS services.

I'll try to get this patch working with master and test it with our code. Happy to contribute it back if of interest, let me know.

[ottokruse]

Maybe very simple/dumb but I'd be happy if every method call would just have an async variant, that would do this under the hood:

# async variant of get()
def get_async(self, *args, executor=None, **kwargs):
    return loop.run_in_executor(
        executor, lambda: self.get(*args, **kwargs) # Call sync version
    )

Saves me a lot of typing as I'm now wrapping every pynamodb call that goes to DynamoDB with loop.run_in_executor myself. (Could of course find more clever ways, e.g. use __getattr__ to dynamically find the sync version to wrap)