python support for asyncio

Hi guys.

Do you consider providing this client library with support for asyncio ?

Regards, Tomasz Prus

(copy of https://github.com/kubernetes-client/python-base/issues/27)

Aug 18 '17 23:08 tomplus

I've prepared the Pull request where I show how callbacks can be used with asyncio. Let me know what you think about it. Thanks.

Aug 19 '17 00:08 tomplus

I just experimented with Swagger v2.3.1 which supports asyncio+aiohttp based client library generation, so that we could use a "native" async interface instead of interpolating callbacks to futures.

What I have done to make it working, starting from the kubernetes-client/gen repository:

Set SWAGGER_CODEGEN_COMMIT in openapi/python.sh to "v2.3.1"
Add <library>asyncio</library> under openapi/python.xml's <configuration> element.

Run the generator script as instructed.

my config was:

export KUBERNETES_BRANCH="release-1.8"
export CLIENT_VERSION="1.0"  # just a random value
export PACKAGE_NAME="kubernetes"

After generation, restructure the generated client library to match the structure guide
— UPDATE: these are the "small changes" in @tomplus comment in #324.
- Relocate the generated kubernetes python package to kubernetes/client.
- Add python-base as kubernetes/base git submodule.
- Create symbolic links kubernetes/stream, kubernetes/config, kubernetes/watch like this repo
- Create kubernetes/__init__.py like this repo
- Replace all absolute imports to use relative imports (e.g., from kubernetes.api_client.apps_v1beta1_deployment import AppsV1beta1Deployment in the generated code to from .apps_v1beta1_deployment import AppsV1beta1Deployment)
- Merge requirements of python-base and the generated library
- pip install -e . inside a virtualenv
- Fix up circular import errors related to V1beta1JSONSchemaProps
- NOTE: I think find and sed commands in the generator's python.sh script should do the above changes automatically, but it didn't work out of the box. So I did manually.
Comment out ssl_verify argument in kubernetes/client/api_client.py since aiohttp forbids passing both ssl_context and ssl_verify option (the later is included in the former)

Then I could run the following example:

import asyncio
from kubernetes import client, config

loop = asyncio.get_event_loop()

async def do():
    config.load_kube_config()
    v1 = client.CoreV1Api()
    ret = await v1.list_pod_for_all_namespaces(watch=False)
    for i in ret.items:
        print("%s\t%s\t%s" % (i.status.pod_ip, i.metadata.namespace, i.metadata.name))

loop.run_until_complete(do())
loop.close()

So the majority of APIs can be converted to use asyncio + aiohttp without much efforts, thanks to Swagger. I hope my "manual" fix-ups could be automated somehow! Still, the python-base streaming wrappers should be rewritten from scratch.

Jan 21 '18 14:01 achimnol

asyncio+aiohttp support would be really appreciated !

Feb 01 '18 14:02 seeker89

I use aiohttp as a server which accesses the python-client so it will be very benefitial to me as well.

Mar 07 '18 07:03 EvgenyKhaliper

Here's how you can do an async watch with the using client (minus error handling):

class Connector(object):
    def __init__(self, q: asyncio.Queue, loop: asyncio.AbstractEventLoop, timeout: int = 30):
        client_config = type.__call__(Configuration)
        config.load_kube_config(client_configuration=client_config, persist_config=False)
        client = ApiClient(configuration=client_config)
        self.__configuration = client_config
        self.__v1 = CoreV1Api(client)
        self.__extv1beta1 = ExtensionsV1beta1Api(client)
        self.__queue = q
        self.__loop = loop

    async def start(self):
        async with aiohttp.ClientSession() as session:
            await asyncio.gather(
                self.__watch(session=session, api=self.__v1, url='/api/v1/endpoints', kind='V1Endpoints'),
                self.__watch(session=session, api=self.__v1, url='/api/v1/services', kind='V1Service'),
                self.__watch(session=session, api=self.__extv1beta1, url='/apis/extensions/v1beta1/ingresses', kind='V1beta1Ingress'),
                loop=self.__loop)

    async def __watch(self, session: aiohttp.ClientSession, api, url: str, kind: str):
        params = {'watch': 'true'}
        async with session.get(self.__configuration.host + url, params=params,
                               headers=self.__configuration.api_key) as response:
            while True:
                chunk = await response.content.readline()
                if not chunk:
                    break
                j = json.loads(chunk)
                j['object'] = api.api_client._ApiClient__deserialize(data=j['object'], klass=kind)
                await self.add_event(j)

May 18 '18 15:05 hanikesn

At last I’ve prepared the new library kubernetes_asyncio which is based on this Python client, but uses the asyncio generator from swagger-codegen. I've added a script to create this client to kubernetes-client/gen repo in this PR https://github.com/kubernetes-client/gen/pull/60. I've also decided to incorporate functionality from sub-repository kubernetes-client/python-base because a lot of changes according to asyncio were needed.

Please take a look, thanks.

May 20 '18 21:05 tomplus

Inspired by @tomplus I have followed in his footsteps and created aiokubernetes.

Unlike kubernetes_asyncio, it is not backwards compatible. The user visible changes are actually minor but allowed me to remove unused Python 2.x code paths and drop Python 3.5 in favor of Python 3.7 (perhaps relevant for #558).

I added documentation and usage examples and would be grateful for any feedback - thank you.

Jul 07 '18 12:07 olitheolix

Please see https://github.com/kubernetes-client/python/pull/324#issuecomment-408554991 for discussion about experimenting asyncio library in this repo.

@olitheolix we need to be backwards compatible for this package. I haven't seen reasons for dropping Python 3.5 (and even strong needs for fully supporting 3.7) yet. What has been changed?

Jul 27 '18 22:07 roycaihw

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

Apr 24 '19 21:04 fejta-bot

/remove-lifecycle stale

Apr 24 '19 21:04 mitar

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

Jul 23 '19 22:07 fejta-bot

/remove-lifecycle stale

Jul 23 '19 22:07 mitar

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

Oct 21 '19 22:10 fejta-bot

/remove-lifecycle stale

Oct 22 '19 05:10 mitar

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

Jan 20 '20 06:01 fejta-bot

/remove-lifecycle stale

Jan 20 '20 06:01 mitar

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

Apr 19 '20 07:04 fejta-bot

/remove-lifecycle stale

Apr 19 '20 15:04 mitar

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

Jul 18 '20 16:07 fejta-bot

/remove-lifecycle stale

Jul 18 '20 16:07 bukowa

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

Oct 16 '20 17:10 fejta-bot

/remove-lifecycle stale

Oct 17 '20 08:10 mindw

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

Jan 15 '21 08:01 fejta-bot

/remove-lifecycle stale

Jan 15 '21 12:01 zegelin

Just curious but, @tomplus what was your motivation for doing this?

Can't sync libraries be used drop-in inside an async application/context?

Mar 19 '21 11:03 mecampbellsoup

I'm not Tom, but I used kubernetes_asyncio in the past to build a highly stateful nginx plus controller based on watches, which requires quite a bit of coordination between different parts of the system. Having a native async library makes this task a lot easier.

Mar 19 '21 12:03 hanikesn

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

Jun 17 '21 12:06 fejta-bot

/remove-lifecycle stale

Jun 17 '21 19:06 mitar

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

Sep 15 '21 20:09 k8s-triage-robot

/remove-lifecycle stale

Sep 15 '21 20:09 remram44

python python copied to clipboard

support for asyncio

python
python copied to clipboard