Kazoo for asyncio

[tailhook]

Hi,

I need an zookeeper protocol implementation for asyncio. While it seems kazoo supports different asynchronous APIs it relies on python stack in a way that doesn't allow integration with asyncio, which needs yield from on every suspension point. So:

1. Any plans to port kazoo on asyncio?
2. Will this kind of patch be accepted?
3. Any recommendations on doing such patch? (it's not enough to just subclass a handler)

[bbangert]

1. Not that I'm aware of
2. Yes.
3. I'd recommend starting with a patch to add it, and stick with a handler subclass as long as possible, on the remaining portion document what needs to change and why, and its effect on the normal Python threading handler.

[tailhook]

I'd recommend starting with a patch to add it, and stick with a handler subclass as long as possible, on the remaining portion document what needs to change and why, and its effect on the normal Python threading handler.

The problem that is every method like start or get of a client must be a coroutine and call it's async counterpart with yield from. I think Client may be split into AsyncClient which is common for both versions, and a thin wrapper which does .get() and all the checking, which will be different for threading and asyncio versions. What do you think?

Also do you think it's reasonable to submit pull requests for small parts of the code, adapting various things to asyncio, before the real handler will be ready? I.e. I need to refactor Retry to yield opcodes rather than calling functions directly, and do aforementioned split of Client class. Or one big patch is better for you?

[harlowja]

How will older version of python work with this change?

• https://pypi.python.org/pypi/trollius (could be useful)

[tailhook]

How will older version of python work with this change?

You will not be able to import the module (because of yield from)

The hack like trollius do, is much slower than native yield from.

[harlowja]

Cool, be interesting to see how this would work out.

[fried]

you could make a asyncio handler that returned asyncio future subclass async result.

As long as you only call the async method names you can yield from them because they are futures :P

[fried]

There is no reason to use yield from inside kazoo distributed code, just use the callback interface of asyncio. That way trollius people can also use it.

[bbangert]

I'm not sure this can be done while maintaining Python 2 compatibility without performance issues. Maybe a hacky workaround might do something like....

1. Spin up a concurrent.futures ThreadPool executer
2. Submit a function to it that starts a kazoo client (and returns a future to you)
3. Make all zkcalls with the client via the threadpool executer

Until we can abandon Python 2 entirely this seems the most reasonable way to use kazoo under async environments that can't yield inline as gevent's can. Maybe I should write up some docs on how to do this as its not too bad in practice (I do lots of calls like this using tornado + Python 3.4 + ThreadPoolExecuter).

Also, when it comes to things like watches and such, you'd have to make special funcs to 'bridge' from the thread back to the async loop by setting a result on a shared future object or something.

[rahulpaul]

Any update on this ?

[tonyseek]

IMO, the threading handler could work well in Python 3 without gevent. It will not be a performance issue because there is only one background threading to keep the state of ZooKeeper session, and it never spawns again in the same client.

We could have a thin wrapper to make methods like client.xxx_async into asyncio coroutine. It is a cheap way without changing the major code of Kazoo.

[bbangert]

I'd like to move to using asyncio as it would drastically simplify a lot of issues in kazoo. I don't think it makes sense to move to asyncio without dropping Python 2 support and I'd like to understand the versions in use by kazoo users more fully before dropping it entirely. Trollius diverges a bit from async in Python 3, and there's the issue that async in Python 3 itself undergoes changes from 3.4 -> 3.4.1 -> 3.4.2 -> 3.5, which is rather annoying.

I've already backported Python 3.5 async functions to work in 3.4.1 once, it was miserable and I wouldn't recommend it to anyone. An alternative would be a kazoo branch that goes Python 3.5+ only, so we could use nice async/await syntax with all the asyncio things.

[fried]

Trollius is a dead project, the author has abandoned it as un-workable.

On Thu, Jun 1, 2017 at 3:15 PM Ben Bangert [email protected] wrote:

I'd like to move to using asyncio as it would drastically simplify a lot of issues in kazoo. I don't think it makes sense to move to asyncio without dropping Python 2 support and I'd like to understand the versions in use by kazoo users more fully before dropping it entirely. Trollius diverges a bit from async in Python 3, and there's the issue that async in Python 3 itself undergoes changes from 3.4 -> 3.4.1 -> 3.4.2 -> 3.5, which is rather annoying.

I've already backported Python 3.5 async functions to work in 3.4.1 once, it was miserable and I wouldn't recommend it to anyone. An alternative would be a kazoo branch that goes Python 3.5+ only, so we could use nice async/await syntax with all the asyncio things.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/python-zk/kazoo/issues/185#issuecomment-305607706, or mute the thread https://github.com/notifications/unsubscribe-auth/AAMPCdKYUlIVAy8NpC6yReuEr80Whxfqks5r_xvkgaJpZM4BtUpJ .

[bbangert]

@fried ah, thanks, I thought I heard something like that.

[harlowja]

Ya, IMHO the whole async stuff is yet-another-rift that the upstream python developers caused (without even calling it python 4.0) and I'm not even sure how many people/developers see that yet. I'd also be in favor of a branch that goes 3.5+ only. It'd be easier (imho) to make that work than trying to shoe-horn both systems under the same umbrella (likely badly).