kubernetes_asyncio

kubernetes_asyncio copied to clipboard

ClientPayloadError occasionally thrown when iterating a watch

[{"_id":"63138fa293201c528473ca22","body":"Is it self-hosted cluster or provided by Google\/AWS\/Azure? Could you check if `kubectl` works longer without the issue?","issue_id":1659922407679,"origin_id":907690960,"user_origin_id":1572228,"create_time":1630185549,"update_time":1630185549,"id":1662226338138,"updated_at":"2022-09-03T17:32:18.138000Z","created_at":"2022-09-03T17:32:18.138000Z"},{"_id":"63138fa293201c528473ca23","body":"this is on a self-hosted microk8s cluster. are you suggesting i try running the same watch with `kubectl`? i didn't know kubectl could perform watches. could you point me to some docs if so?\r\n\r\nwith thanks","issue_id":1659922407679,"origin_id":907703006,"user_origin_id":3240247,"create_time":1630193801,"update_time":1630193801,"id":1662226338141,"updated_at":"2022-09-03T17:32:18.141000Z","created_at":"2022-09-03T17:32:18.141000Z"},{"_id":"63138fa293201c528473ca24","body":"There is a flag `--watch` to watch for changes, for example:\r\n\r\n `kubectl get pods --watch`","issue_id":1659922407679,"origin_id":908070360,"user_origin_id":1572228,"create_time":1630305226,"update_time":1630305226,"id":1662226338143,"updated_at":"2022-09-03T17:32:18.143000Z","created_at":"2022-09-03T17:32:18.143000Z"},{"_id":"63138fa293201c528473ca25","body":"righto, i'll take a look. regardless, should the `aiohttp.client_exceptions.ClientPayloadError` be wrapped? (i'm not sure, i can see arguments both ways)","issue_id":1659922407679,"origin_id":908071761,"user_origin_id":3240247,"create_time":1630305399,"update_time":1630305399,"id":1662226338145,"updated_at":"2022-09-03T17:32:18.145000Z","created_at":"2022-09-03T17:32:18.145000Z"},{"_id":"63138fa293201c528473ca26","body":"How long is it working without raising the exception?\r\n\r\nI'm also not sure what to do... we can treat it as \"timeout\" and silently reconnect but on the other hand such behavior may hide some errors in other case.","issue_id":1659922407679,"origin_id":908727341,"user_origin_id":1572228,"create_time":1630360424,"update_time":1630360424,"id":1662226338147,"updated_at":"2022-09-03T17:32:18.147000Z","created_at":"2022-09-03T17:32:18.147000Z"},{"_id":"63138fa293201c528473ca27","body":"> Could you check if kubectl works longer without the issue?\r\n\r\ni haven't tested exhaustively, but it doesn't look like kubectl lasts longer.\r\n\r\n> How long is it working without raising the exception?\r\n\r\nit works for quite some time. i think this might typically result in about 4 pod restarts in 24 hours.\r\n\r\n> I'm also not sure what to do... we can treat it as \"timeout\" and silently reconnect but on the other hand such behavior may hide some errors in other case.\r\n\r\nyeah. ideally we can clear tell between exceptions we expect, and those we don't. something like:\r\n\r\n```\r\nwhile True:\r\n try:\r\n async with watch.stream(self._api.list_namespaced_pod, self._namespace) as stream:\r\n async for update in stream:\r\n except ExceptionsWeExpect:\r\n pass\r\n except ExceptionsWeDontExpect:\r\n raise\r\n```\r\n\r\nwith thanks","issue_id":1659922407679,"origin_id":921444804,"user_origin_id":3240247,"create_time":1631851946,"update_time":1631851946,"id":1662226338149,"updated_at":"2022-09-03T17:32:18.149000Z","created_at":"2022-09-03T17:32:18.149000Z"},{"_id":"63138fa293201c528473ca29","body":"We are seeing this issue in an Azure kubernetes 1.22 cluster. Whenever we watch anything and there is no activity for the 5 minute default kubernetes timeout (even if prior events have been received), we see this error.\r\n\r\nI am not familiar (enough) with the underpinnings of how kubernetes does watch calls, but I could run a `kubectl get pods --watch` on the same cluster and it worked past the 5 minute timeout.","issue_id":1659922407679,"origin_id":1187292833,"user_origin_id":1258576,"create_time":1658147545,"update_time":1658147545,"id":1662226338152,"updated_at":"2022-09-03T17:32:18.152000Z","created_at":"2022-09-03T17:32:18.152000Z"}]

i do get a ClientPayloadError from time to time: ``` File ... in run async for update in stream: File "/usr/local/lib/python3.8/dist-packages/kubernetes_asyncio/watch/watch.py", line 131, in __anext__ return await self.next() File "/usr/local/lib/python3.8/dist-packages/kubernetes_asyncio/watch/watch.py",...

jonathon-love

dynamic.DynamicClient

[{"_id":"63138fa293201c528473ca2a","body":"Yes, it's on my TODO list.","issue_id":1659922407682,"origin_id":1158191468,"user_origin_id":1572228,"create_time":1655418409,"update_time":1655418409,"id":1662226338449,"updated_at":"2022-09-03T17:32:18.448000Z","created_at":"2022-09-03T17:32:18.448000Z"},{"_id":"6605bf0daa97b6ed0b057ed1","body":"@tomplus if this can be a port of the existing code in the sync client I'd be glad to work on it. ","issue_id":1659922407682,"origin_id":1531401995,"user_origin_id":4935304,"create_time":1683030928,"update_time":1683030928,"id":1711652621791,"updated_at":"2024-03-28T19:03:41.790000Z","created_at":"2024-03-28T19:03:41.790000Z"},{"_id":"6605bf0daa97b6ed0b057ed2","body":"@bobh66 Thanks! You can port code from the sync client and make it asynchronous. Let me know if you have any questions.","issue_id":1659922407682,"origin_id":1531631516,"user_origin_id":1572228,"create_time":1683039493,"update_time":1683039493,"id":1711652621797,"updated_at":"2024-03-28T19:03:41.797000Z","created_at":"2024-03-28T19:03:41.797000Z"},{"_id":"6605bf0daa97b6ed0b057ed3","body":"I think the main question is how to handle the async initialization requirements of the Discoverer and DynamicClient classes? They both need to do async operations on class creation, so some sort of factory method is needed, like:\r\n\r\n```\r\nclass Discoverer:\r\n ......\r\n async def _ainit(self):\r\n await self.__init_cache()\r\n\r\n @classmethod\r\n async def create(cls, client, cache_file=None):\r\n self = cls(client=client, cache_file=cache_file)\r\n await self._ainit()\r\n return self\r\n```\r\n\r\nand then the class creation becomes:\r\n\r\n```\r\nself.__discoverer = discoverer.create(self, self.cache_file)\r\n```\r\n\r\nIs there a better pattern or is this an acceptable API for the classes?\r\n\r\nThanks\r\n\r\n","issue_id":1659922407682,"origin_id":1531646858,"user_origin_id":4935304,"create_time":1683040057,"update_time":1683040057,"id":1711652621801,"updated_at":"2024-03-28T19:03:41.800000Z","created_at":"2024-03-28T19:03:41.800000Z"},{"_id":"6605bf0daa97b6ed0b057ed4","body":"I like your solution, you can implemented it in this way :+1: ","issue_id":1659922407682,"origin_id":1531697746,"user_origin_id":1572228,"create_time":1683042097,"update_time":1683042097,"id":1711652621803,"updated_at":"2024-03-28T19:03:41.802000Z","created_at":"2024-03-28T19:03:41.802000Z"},{"_id":"6605bf0daa97b6ed0b057ed5","body":"I was able to come up with a better solution using the `__await__` magic method so that the DynamicClient and Discoverer objects can be await'ed directly and a separate create method is not needed.\r\n\r\nThese changes are ready to be pushed I'm just waiting on some process overhead from my employer and then I'll create a PR.","issue_id":1659922407682,"origin_id":1549956442,"user_origin_id":4935304,"create_time":1684253004,"update_time":1684253020,"id":1711652621805,"updated_at":"2024-03-28T19:03:41.805000Z","created_at":"2024-03-28T19:03:41.805000Z"},{"_id":"6605bf0daa97b6ed0b057ed6","body":"@bobh66 Thanks for this! Looking forward to using it.","issue_id":1659922407682,"origin_id":1607164094,"user_origin_id":684915,"create_time":1687774893,"update_time":1687774893,"id":1711652621812,"updated_at":"2024-03-28T19:03:41.811000Z","created_at":"2024-03-28T19:03:41.811000Z"}]

Any plan to have dynamic.DynamicClient ?

devlounge

Custom object json-patch

[{"_id":"63138fa293201c528473ca2b","body":"As you checked, patching custom objects supports `application\/merge-patch+json` only so I guess API Server (or CRD) rejects a request with different type of merge, doesn't it?","issue_id":1659922407686,"origin_id":499489198,"user_origin_id":1572228,"create_time":1559827003,"update_time":1559827003,"id":1662226338850,"updated_at":"2022-09-03T17:32:18.849000Z","created_at":"2022-09-03T17:32:18.849000Z"},{"_id":"63138fa293201c528473ca2c","body":"When I tried to patch with json patch on a custom object using kubectl it worked, so assuming that the api server handles it. We should leave out the strategic patch since that does not make sense for a custom object, my wrong, at least until strategic patching functionality can be specified in a CRD. \r\n\r\nSo I think that both json and merge patch should be specified in the swagger file. \r\n","issue_id":1659922407686,"origin_id":499494058,"user_origin_id":3320413,"create_time":1559827772,"update_time":1559827772,"id":1662226338854,"updated_at":"2022-09-03T17:32:18.853000Z","created_at":"2022-09-03T17:32:18.853000Z"},{"_id":"63138fa293201c528473ca2d","body":"Fixed in the latest release - v10.0.0","issue_id":1659922407686,"origin_id":531995711,"user_origin_id":1572228,"create_time":1568676312,"update_time":1568676312,"id":1662226338857,"updated_at":"2022-09-03T17:32:18.856000Z","created_at":"2022-09-03T17:32:18.856000Z"},{"_id":"63138fa293201c528473ca2e","body":"Hi, seems to have got some issues now.\r\n\r\n Reason: Unsupported Media Type\r\n HTTP response headers: <CIMultiDictProxy('Content-Type': 'application\/json', 'Date': 'Thu, 19 Sep 2019 12:24:53 GMT', 'Content-Length': '263')>\r\n HTTP response body: {\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"the body of the request was in an unknown format - accepted media types include: application\/json-patch+json, application\/merge-patch+json\",\"reason\":\"UnsupportedMediaType\",\"code\":415}\r\n\r\nGenerated `custom_objects_api.py` have:\r\n\r\n def patch_namespaced_custom_object_with_http_info(self, group, version, namespace, plural, name, body, **kwargs):\r\n ...\r\n header_params['Content-Type'] = self.api_client.select_header_content_type( # noqa: E501\r\n ['application\/json-patch+json', 'application\/merge-patch+json']) # noqa: E501\r\n\r\n`select_header_content_type`returns first i.e. `'application\/json-patch+json'`\r\n\r\nThen in `rest.py`we have:\r\n\r\n if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']:\r\n if re.search('json', headers['Content-Type'], re.IGNORECASE):\r\n if headers['Content-Type'] == 'application\/json-patch+json':\r\n if not isinstance(body, list):\r\n headers['Content-Type'] = 'application\/strategic-merge-patch+json'\r\n\r\nWhen patching, Content-Type contains 'json' and now since content type is `'application\/json-patch+json'` is true it checks if body not a list which is also true, leads to that the content type is set to `'application\/strategic-merge-patch+json'`, which is not supported by custom objects APIs. The api-server returns unsupported media type (415).\r\n\r\nThe last setting of content type would need to be `'application\/merge-patch+json'`\r\n\r\nSo best solution would be if the generated code would select the correct content type directly, based on method, body is list and allowed content types. Would that be possible? Alternatively, the `select_header_content_type` should return a list when it can't select and then allow the `request` in `rest.py` to do the proper selection and replace the content type list with a specific string.\r\n\r\nWhat do you think @tomplus ?\r\n\r\n","issue_id":1659922407686,"origin_id":533138086,"user_origin_id":3320413,"create_time":1568900775,"update_time":1568900804,"id":1662226338858,"updated_at":"2022-09-03T17:32:18.858000Z","created_at":"2022-09-03T17:32:18.858000Z"},{"_id":"63138fa293201c528473ca2f","body":"Thanks, definitely it needs to be fixed.","issue_id":1659922407686,"origin_id":533746017,"user_origin_id":1572228,"create_time":1569023258,"update_time":1569023258,"id":1662226338860,"updated_at":"2022-09-03T17:32:18.860000Z","created_at":"2022-09-03T17:32:18.860000Z"},{"_id":"63138fa293201c528473ca30","body":"Hi, I'm running into this issue now. Any quick way of working around the \"media type not supported\" error while your fix is not yet merged?","issue_id":1659922407686,"origin_id":661847097,"user_origin_id":10549675,"create_time":1595336535,"update_time":1595336535,"id":1662226338863,"updated_at":"2022-09-03T17:32:18.862000Z","created_at":"2022-09-03T17:32:18.862000Z"},{"_id":"63138fa293201c528473ca31","body":"Could you tell more about your use case? Do you try to use different merge strategy?","issue_id":1659922407686,"origin_id":662146414,"user_origin_id":1572228,"create_time":1595371878,"update_time":1595371878,"id":1662226338865,"updated_at":"2022-09-03T17:32:18.864000Z","created_at":"2022-09-03T17:32:18.864000Z"},{"_id":"63138fa293201c528473ca32","body":"I'm just trying to patch a custom object:\r\n```\r\napi_response = await api_instance.patch_namespaced_custom_object(group, version, manifest['metadata'].get('namespace'), plural, manifest['metadata']['name'], manifest)\r\n```\r\n\r\nBut it's throwing me an error on the Content Type:\r\n\r\n```\r\n~ Failed to deploy the manifest for HelmRelease tmp -> (415)\r\nReason: Unsupported Media Type\r\nHTTP response headers: <CIMultiDictProxy('Audit-Id': '75d76ce6-7de6-432f-a5d9-302ee6790a16', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application\/json', 'Date': 'Tue, 21 Jul 2020 12:43:59 GMT', 'Content-Length': '293')>\r\nHTTP response body: {\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"the body of the request was in an unknown format - accepted media types include: application\/json-patch+json, application\/merge-patch+json, application\/apply-patch+yaml\",\"reason\":\"UnsupportedMediaType\",\"code\":415}\r\n```\r\n\r\nAll I want to do is to patch the resource whith the changes coming in, that used to work fine when I used the default library.\r\nIs there a different way I should be calling this method?\r\n\r\nThe manifest I'm sending is rather simple:\r\n```\r\n {\r\n \"apiVersion\": \"helm.fluxcd.io\/v1\",\r\n \"kind\": \"HelmRelease\",\r\n \"metadata\": {\r\n \"generation\": 1,\r\n \"name\": \"tmp\",\r\n \"namespace\": \"tmp\"\r\n },\r\n \"spec\": {\r\n \"chart\": {\r\n \"name\": \"tmp\",\r\n \"repository\": \"https:\/\/<redacted>\/repo\",\r\n \"version\": \"0.1.1\"\r\n },\r\n \"helmVersion\": \"v3\",\r\n \"releaseName\": \"tmp\",\r\n \"values\": {}\r\n }\r\n }\r\n```","issue_id":1659922407686,"origin_id":662430643,"user_origin_id":10549675,"create_time":1595421765,"update_time":1595421765,"id":1662226338867,"updated_at":"2022-09-03T17:32:18.867000Z","created_at":"2022-09-03T17:32:18.867000Z"},{"_id":"63138fa293201c528473ca33","body":"Thanks. As a workaround you can try to send list with [JSON Patch](http:\/\/jsonpatch.com\/) instructions to modify your CRD. This nice library https:\/\/github.com\/stefankoegl\/python-json-patch creates such patches from differences between dicts.","issue_id":1659922407686,"origin_id":663913994,"user_origin_id":1572228,"create_time":1595717575,"update_time":1595717575,"id":1662226338869,"updated_at":"2022-09-03T17:32:18.869000Z","created_at":"2022-09-03T17:32:18.869000Z"},{"_id":"63138fa293201c528473ca34","body":"Thanks @tomplus ! That did work. I had to make sure to remove the patches I didn't want as the new manifest would not have the UID and the other information from the existing resource:\r\n```\r\nrequest_body = jsonpatch.make_patch(api_response, new_manifest)\r\nrequest_patch = [patch for patch in request_body if patch['path'] != \"\/metadata\/resourceVersion\" and patch['path'] != \"\/metadata\/uid\" and patch['path'] != \"\/status\" and patch['path'] != \"\/metadata\/selfLink\" and patch['path'] != \"\/metadata\/creationTimestamp\" and patch['path'] != \"\/metadata\/generation\"] \r\n```\r\n`api_response` is the response from the K8s api with the current resource.\r\n`new_manifest` is the new manifest with the changes.\r\n\r\nIt's now working fine!","issue_id":1659922407686,"origin_id":664433116,"user_origin_id":10549675,"create_time":1595860399,"update_time":1595860399,"id":1662226338871,"updated_at":"2022-09-03T17:32:18.871000Z","created_at":"2022-09-03T17:32:18.871000Z"},{"_id":"63138fa293201c528473ca36","body":"I have also been frustrated by this recently, specifically because the behavior differs from the official Python kubernetes client. There is a discussion here of why custom object patching in kubernetes-client works the way it does:\r\n\r\nhttps:\/\/github.com\/kubernetes-client\/python\/issues\/866\r\n\r\nSo basically, it seems, kubernetes-client forces the caller to use application\/merge-patch+json, where kubernetes_asyncio forces the use of application\/json-patch+json. Both are probably wrong, or at least sub-optimal.\r\n\r\nI don't know if kubernetes_asyncio has a goal of 100% matching the kubernetes-client API, but it is a convenient quality when you are trying to convert from kubernetes-client to kubernetes_asyncio as I am.","issue_id":1659922407686,"origin_id":720584265,"user_origin_id":25060759,"create_time":1604335032,"update_time":1604335032,"id":1662226338874,"updated_at":"2022-09-03T17:32:18.873000Z","created_at":"2022-09-03T17:32:18.873000Z"},{"_id":"63138fa293201c528473ca37","body":"Totally agree, patching in this library should work in the same way as in the kubernetes-client. I wasn't aware of the differences but as you noticed both behavior are problematic. I've started working on fixing both libraries (by extending openapi-generator) and your post gives me additional motivation to end it up. Thanks.","issue_id":1659922407686,"origin_id":720770607,"user_origin_id":1572228,"create_time":1604357642,"update_time":1604357642,"id":1662226338876,"updated_at":"2022-09-03T17:32:18.875000Z","created_at":"2022-09-03T17:32:18.875000Z"},{"_id":"63138fa293201c528473ca38","body":"This issue continues to frustrate me as well. I am able to workaround it for now but just wanted to add my +1\r\n\r\n[kubernetes-client\/python#866](https:\/\/github.com\/kubernetes-client\/python\/issues\/866)","issue_id":1659922407686,"origin_id":760488494,"user_origin_id":19753981,"create_time":1610659797,"update_time":1610659797,"id":1662226338878,"updated_at":"2022-09-03T17:32:18.877000Z","created_at":"2022-09-03T17:32:18.877000Z"},{"_id":"63138fa293201c528473ca39","body":"There is a PR to track https:\/\/github.com\/kubernetes-client\/python\/pull\/959 - it's already prepared and waits for review.","issue_id":1659922407686,"origin_id":760507488,"user_origin_id":1572228,"create_time":1610662181,"update_time":1610662198,"id":1662226338880,"updated_at":"2022-09-03T17:32:18.880000Z","created_at":"2022-09-03T17:32:18.880000Z"},{"_id":"63138fa293201c528473ca3a","body":"First off, a huge thank you to @tomplus for his ongoing work to resolve this issue. In the meantime, I've found a somewhat ugly workaround that can be used in the interim. It turns out these headers can be set using [`ApiClient.set_default_header`](https:\/\/github.com\/tomplus\/kubernetes_asyncio\/blob\/5f4a80b40b5f520b27b808aa0556577b4189166a\/kubernetes_asyncio\/client\/api_client.py#L118) which will [override the headers](https:\/\/github.com\/tomplus\/kubernetes_asyncio\/blob\/5f4a80b40b5f520b27b808aa0556577b4189166a\/kubernetes_asyncio\/client\/api_client.py#L132) that are [set by the CustomObjectsApi](https:\/\/github.com\/tomplus\/kubernetes_asyncio\/blob\/5f4a80b40b5f520b27b808aa0556577b4189166a\/kubernetes_asyncio\/client\/api\/custom_objects_api.py#L2949-L2953). eg. `api_instance.api_client.set_default_header('content-type', 'application\/merge-patch+json')`\r\n\r\n@HaraldGustafsson @gcarrarom @hapatrick I know its been a while since your comments but heads up WRT to the above workaround","issue_id":1659922407686,"origin_id":981999372,"user_origin_id":19753981,"create_time":1638218109,"update_time":1638218109,"id":1662226338882,"updated_at":"2022-09-03T17:32:18.881000Z","created_at":"2022-09-03T17:32:18.881000Z"},{"_id":"63138fa293201c528473ca3b","body":"@linkous8 Thanks for your workaround here -- when I used this, I needed to use `\"Content-Type\"` instead of `\"content-type\"` per the naming of the variable here: https:\/\/github.com\/kubernetes-client\/python\/blob\/master\/kubernetes\/client\/api\/custom_objects_api.py#L2427","issue_id":1659922407686,"origin_id":1048195402,"user_origin_id":7889440,"create_time":1645562619,"update_time":1645562636,"id":1662226338885,"updated_at":"2022-09-03T17:32:18.884000Z","created_at":"2022-09-03T17:32:18.884000Z"},{"_id":"6605c2d2aa97b6ed0b057f5e","body":"Starting from v22.6.0 it's possible to call patch methods with forcing a content type. In a next big release (v30.x) there is a new version of logic to detect a content-type for a patch. There is also an [example](https:\/\/github.com\/tomplus\/kubernetes_asyncio\/blob\/master\/examples\/patch.py) how to use different type of patch. Changes are already merged to master branch so feel free to test it in the meantime. Thanks.\r\n","issue_id":1659922407686,"origin_id":1972002971,"user_origin_id":1572228,"create_time":1709242573,"update_time":1709242573,"id":1711653586949,"updated_at":"2024-03-28T19:19:46.949000Z","created_at":"2024-03-28T19:19:46.949000Z"}]

I noticed that most patch calls support all 3 patch types (json-patch, merge-patch, strategic-merge-patch), but the patch calls for custom objects (including for status and scale sub-resources) only support merge-patch....

HaraldGustafsson

Perform automatic retries for 410s during watches

[{"_id":"63138fa393201c528473ca3c","body":"I'm not sure I still agree with the change proposed originally. I'm going to think about it some more, and will update this issue once I have a better idea of what the desired behavior is (see discussion in the see also links).","issue_id":1659922407689,"origin_id":778933519,"user_origin_id":144119,"create_time":1613364223,"update_time":1613364223,"id":1662226339196,"updated_at":"2022-09-03T17:32:19.196000Z","created_at":"2022-09-03T17:32:19.196000Z"},{"_id":"63138fa393201c528473ca3d","body":"I ran into this. I am trying to fix the bug, but it is hard to validate a fix since it takes a long time before a 410 gets returned by Kubernetes. An automatic retry in kubernetes-asyncio would be appreciated, especially since the sync kubernetes client has it.","issue_id":1659922407689,"origin_id":1031030701,"user_origin_id":426784,"create_time":1644203737,"update_time":1644203737,"id":1662226339200,"updated_at":"2022-09-03T17:32:19.200000Z","created_at":"2022-09-03T17:32:19.200000Z"}]

At this time, when a watch expires (with a 410 Gone) the watcher returns an "error" event to the caller. Instead of returning a raw error event, it should attempt...

JacobHenner

stream doesn't support multichannel communication

[{"_id":"63138fa393201c528473ca3e","body":"I know that i will close the issues... \r\nhttps:\/\/github.com\/tomplus\/kubernetes_asyncio\/issues\/12","issue_id":1659922407692,"origin_id":493308345,"user_origin_id":16299123,"create_time":1558064578,"update_time":1558064608,"id":1662226339510,"updated_at":"2022-09-03T17:32:19.509000Z","created_at":"2022-09-03T17:32:19.509000Z"},{"_id":"63138fa393201c528473ca3f","body":"Oof. This is biting me as I try to get the test suite working for a kubernetes_asyncio-based JupyterHUb Kubespawner. I'll see if I can figure any of that out.","issue_id":1659922407692,"origin_id":1005029217,"user_origin_id":2453165,"create_time":1641317763,"update_time":1641317763,"id":1662226339513,"updated_at":"2022-09-03T17:32:19.512000Z","created_at":"2022-09-03T17:32:19.512000Z"},{"_id":"6605bde4d32ed319a500afb9","body":"@tomplus - I'm finishing up the DynamicClient work for #209 hopefully this week, and then I'll probably take a shot at this, or at least dig into it and see how much work it is.","issue_id":1659922407692,"origin_id":1543040580,"user_origin_id":4935304,"create_time":1683768099,"update_time":1683768099,"id":1711652324908,"updated_at":"2024-03-28T18:58:44.908000Z","created_at":"2024-03-28T18:58:44.908000Z"},{"_id":"6605bde4d32ed319a500afba","body":"I have a working implementation of stream() - portforward() is a bit more complicated and I'm still working on that.","issue_id":1659922407692,"origin_id":1549936994,"user_origin_id":4935304,"create_time":1684252409,"update_time":1684252409,"id":1711652324915,"updated_at":"2024-03-28T18:58:44.914000Z","created_at":"2024-03-28T18:58:44.914000Z"}]

Current implementation of web-socket doesn't support multichannel communication. Now it's impossible to "talk" to a pod like sending a command, reading output, send another command ... in the same connection.

tomplus

Malformed JSON response, the 'object' and/or 'type' field is missing in JSON

[{"_id":"63138fa393201c528473ca40","body":"Thanks for reporting it.","issue_id":1659922407695,"origin_id":985008522,"user_origin_id":1572228,"create_time":1638479889,"update_time":1638479889,"id":1662226339812,"updated_at":"2022-09-03T17:32:19.811000Z","created_at":"2022-09-03T17:32:19.811000Z"}]

we maintaining this client's fork @ https://github.com/truenas/kubernetes_asyncio – I have faced an issue recently where client is unable to parse out 400 error and instead raise a bare exception. Steps...

Qubad786

resource_version set incorrectly while watching list_* functions

[{"_id":"63138fa493201c528473ca41","body":"Hi @JacobHenner \r\n\r\nUsing the resource version was also discussed in the official library: https:\/\/github.com\/kubernetes-client\/python\/issues\/819 \r\n\r\nWe can use the resource version as you suggest and implement the list-and-watch pattern but the question is if we can use a resVer from arrived events to implement list-watch-watch-... pattern?\r\n\r\nIMO bookmarks can be use to some optimization only. Watch should work even if API Server doesn't support it.\r\n\r\n410 - definitely has to be ported from the official repository. Are you going to work on it?\r\n\r\nThanks for reported issues :+1: ","issue_id":1659922407697,"origin_id":778704128,"user_origin_id":1572228,"create_time":1613265478,"update_time":1613265478,"id":1662226340146,"updated_at":"2022-09-03T17:32:20.146000Z","created_at":"2022-09-03T17:32:20.146000Z"},{"_id":"63138fa493201c528473ca42","body":"I'll try to look at these three issues tomorrow. \r\n\r\n> We can use the resource version as you suggest and implement the list-and-watch pattern but the question is if we can use a resVer from arrived events to implement list-watch-watch-... pattern?\r\n\r\nI'll take a look at the Go library to see how it is handled there. Perhaps that'll be helpful.\r\n\r\n> IMO bookmarks can be use to some optimization only. Watch should work even if API Server doesn't support it.\r\n\r\nIndeed, I will keep this in mind.","issue_id":1659922407697,"origin_id":778707066,"user_origin_id":144119,"create_time":1613267315,"update_time":1613267315,"id":1662226340149,"updated_at":"2022-09-03T17:32:20.149000Z","created_at":"2022-09-03T17:32:20.149000Z"},{"_id":"63138fa493201c528473ca43","body":"So after conducting some more research[1], it seems as if the client-go library maintains a distinction between watches and informers. The former is the low-level operation, the latter is a high-level abstraction that wraps a watch and provides a cache, handles watcher reconnection, etc. This concept does not exist in the OpenAPI-derived libraries by default, although some of them (such as the [go](https:\/\/github.com\/kubernetes-client\/go\/pull\/28) one) have decided to implement a similar feature.\r\n\r\nFor Python, there is an open issue for supporting informers in the official library: https:\/\/github.com\/kubernetes-client\/python\/issues\/868. It seems to be stalled, and there's an outstanding question of whether higher-level concepts belong in the OpenAPI-derived libraries (or if they should be packaged separately).\r\n\r\nTo resolve this issue, I first propose:\r\n\r\n1. The internal resource_version of the watcher object is set to the resource_version of the first returned list. This will prevent reconnection issues when timeouts occur, as long as the resource_version is still valid.\r\n\r\nThen, we consider if we should:\r\n\r\n1. Close #136 without addressing it at the watcher level, and let callers implement the higher-level behavior they need. Once the watch receives the 410 Gone response, retrying it with the same resource_version seems unproductive. Looking at it again, I'm unsure what the PR I linked to in #136 accomplished - maybe it was an attempt to accommodate eventual consistency? It looks like one of the reviewers [had concerns](https:\/\/github.com\/kubernetes-client\/python-base\/pull\/133#discussion_r330247229) of whether this type of retry should be built-in.\r\n2. Handle #136 partially, by adding support for watch bookmarks. If available, the watch's internal resource_version would be set to the resourceVersion specified in bookmark events, and retry with the updated resourceVersion upon receiving a 410 Gone. If no bookmark events were received, the retry would fail, and the caller would need to handle the failure.\r\n3. Close #136 without addressing it at the watcher level, and implement an informer abstraction that would handle watch retries, bookmarks, caching, etc. Determine if this is a feature that should be specific to this async library, or if it's something that should be shared or similar between this library and the official one.\r\n\r\nOnce a decision is made, I'll start working on a PR. Options 1 and 2 would be straightforward to handle. Option 3 would require some additional discussion.\r\n\r\n[1]: https:\/\/www.youtube.com\/watch?v=PLSDvFjR9HY","issue_id":1659922407697,"origin_id":778824511,"user_origin_id":144119,"create_time":1613329236,"update_time":1613329236,"id":1662226340151,"updated_at":"2022-09-03T17:32:20.150000Z","created_at":"2022-09-03T17:32:20.150000Z"},{"_id":"63138fa493201c528473ca44","body":"I've written a small proof-of-concept informer-like wrapper. I have to finish adding some features and testing the implementation before sharing, but perhaps it'd be useful here. So far it has support for bookmarks, reconnections on 410s, and caching. The cache is used to avoid replaying or missing events that occurred between the time a watch went invalid and the time a replacement watch was started. \r\n\r\nAssuming it is useful, I have two follow-up questions:\r\n\r\n1. Would that kind of wrapper be considered for inclusion in this library, or does it belong in a separate library? (This question was asked for informers in the official Python client as well, see link above).\r\n2. If this were to be included, would #134 and #136 still need to be handled in the current implementation, or would the advice be to use the higher-level abstraction unless the user intends to handle 410s and error events on their own?","issue_id":1659922407697,"origin_id":778936229,"user_origin_id":144119,"create_time":1613364749,"update_time":1613364749,"id":1662226340153,"updated_at":"2022-09-03T17:32:20.152000Z","created_at":"2022-09-03T17:32:20.152000Z"},{"_id":"63138fa493201c528473ca45","body":"Informer sounds as a really good idea. It looks it's the only way to handle reconnection, using resource version properly without notifying about the same events. IMO it should be a part of the library as a go-client has it. I'm also not sure if its possible to create one solution for both libraries. I'm afraid it has to be ported\/synced manually from one to another as we port Watch() and other components.\r\n\r\nWe can advice how to deal with reconnection in Watch() using some examples and recommend to use your informer as an efficient way to observe K8s' objects.\r\n\r\nLet me know if I can help you with this feature. Thanks.","issue_id":1659922407697,"origin_id":779145200,"user_origin_id":1572228,"create_time":1613387015,"update_time":1613387015,"id":1662226340155,"updated_at":"2022-09-03T17:32:20.154000Z","created_at":"2022-09-03T17:32:20.154000Z"},{"_id":"63138fa493201c528473ca46","body":"So far I've implemented an informer-like wrapper which handles 410 Gone reconnections, bookmarking, and basic event-handling. I'm still working to figure out how closely to align it with the Informer implementations in the client-go, Java, and OpenAPI go libraries. Both client-go and the Java have complex implementations with many abstractions. I'm not yet sure the extent to which that's necessary here, and I'm thinking about how to make the implementation extensible so it can be published incrementally.\r\n\r\nAs for the original topic presented in this issue, I will provide feedback after thinking about it some more. I believe that some conditions should be handled within the watch itself and other should be handled in higher-level abstractions, but I haven't yet figured out which features fall in which category. When my changes are ready for PR, perhaps the location of some of the reliability measures can be discussed.","issue_id":1659922407697,"origin_id":779483773,"user_origin_id":144119,"create_time":1613429948,"update_time":1613430367,"id":1662226340156,"updated_at":"2022-09-03T17:32:20.156000Z","created_at":"2022-09-03T17:32:20.156000Z"},{"_id":"63138fa493201c528473ca47","body":"Didn't have as much time as I hoped to work on this over the week. Will send something over once ready. My initial implementation has been focused on increasing reliability and providing an informer-like object with similar but not exactly the same semantics as client-go. It's possible that some of the changes introduced should be moved into the watcher itself. I'll comment on the PR in those places once it's available.\r\n\r\nI'm now looking through client-go to determine if I should modify my current implementation to be more similar, or if I should continue as-is. I want to make sure I haven't ignored any intentional stability or \"correctness\" decisions made by the client-go devs.","issue_id":1659922407697,"origin_id":782973785,"user_origin_id":144119,"create_time":1613955802,"update_time":1613955802,"id":1662226340159,"updated_at":"2022-09-03T17:32:20.159000Z","created_at":"2022-09-03T17:32:20.159000Z"},{"_id":"63138fa493201c528473ca48","body":"Apologies for not providing an implementation sooner, my priorities have changed and I haven't had much time to work on this. \r\n\r\nI've published an initial implementation of my idea in a separate project, since I'm not certain of which parts should be included in the library's Watch class, and which should remain separate. It can be reviewed [here](https:\/\/github.com\/JacobHenner\/kubernetes_asyncio_informer\/), with an example [here](https:\/\/github.com\/JacobHenner\/kubernetes_asyncio_informer\/blob\/main\/examples\/watch_configmaps.py).\r\n\r\nI've commented on the sections that might be considered for inclusion in the Watch class. After that's settled, we can determine whether or not the informer belongs within this library, or separate. ","issue_id":1659922407697,"origin_id":821896184,"user_origin_id":144119,"create_time":1618698538,"update_time":1618698538,"id":1662226340161,"updated_at":"2022-09-03T17:32:20.161000Z","created_at":"2022-09-03T17:32:20.161000Z"},{"_id":"63138fa493201c528473ca49","body":"Thanks for your reply.\r\n\r\nI agree with you, the part of your library related to the Watch can be moved here to fix the base library - better reconnect handling, supporting bookmarks etc. The informer implementation can be developed as a separate library.\r\n\r\nLet me know if I can help in any way.","issue_id":1659922407697,"origin_id":831600278,"user_origin_id":1572228,"create_time":1620085007,"update_time":1620085007,"id":1662226340163,"updated_at":"2022-09-03T17:32:20.163000Z","created_at":"2022-09-03T17:32:20.163000Z"}]

Watch objects maintain a resource_version field to allow the watch to resume at the last observed point after a timeout. However, there is a bug preventing this from working as...

JacobHenner

ApiClient.deserialize should be a function, not a method

[{"_id":"63138fa493201c528473ca4a","body":"Hi @horazont \r\n\r\nI'm afraid this refactoring won't be easy. Serialize() calls __serialize() which uses some class attributes: https:\/\/github.com\/tomplus\/kubernetes_asyncio\/blob\/master\/kubernetes_asyncio\/client\/api_client.py#L304 And what's more it's auto-generated by openapi-generator so we have to change the generator first.\r\n\r\nOther solutions which you can consider:\r\n\r\n1. Use context manager with ApiClient or await ApiClient.close() after deserialization.\r\n\r\n2. If it's safe, disable aiohttp warning (configure logging) :smiling_imp: \r\n\r\n3. Use https:\/\/github.com\/kubernetes-client\/python - it's synchronous version of this library\r\n \r\n\r\n\r\n\r\n","issue_id":1659922407700,"origin_id":733996971,"user_origin_id":1572228,"create_time":1606348625,"update_time":1606348625,"id":1662226340477,"updated_at":"2022-09-03T17:32:20.477000Z","created_at":"2022-09-03T17:32:20.477000Z"},{"_id":"63138fa493201c528473ca4b","body":"Aha, I see. I scanned the methods *twice* for references to `self` beyond the calls to `__deserialize_*` and missed that. Sorry.\r\n\r\nYet, this would still work just the same if the methods were `@classmethod`, which had about the same effect. How hard would it be to teach that to the generator? Easier than breaking it out in a separate function\/module I expect.\r\n\r\nAs a workaround, we\u2019re currently doing:\r\n\r\n```python\r\n api_client = ...\r\n try:\r\n # do things\r\n pass\r\n finally:\r\n asyncio.create_task(api_client.close())\r\n```\r\n\r\n:-X\r\n\r\nTerrible enough, but also shuts up the warning. We don\u2019t want to silence it globally because we had cases of api_client leaks in the past and we want to learn about them. We also don\u2019t want to load both kubernetes_asyncio and kubernetes, and we need kubernetes_asyncio for it being asyncio :).\r\n\r\nThe root issue we\u2019re trying to solve is that we need to write a function which inspects V1PodSpec and it needs to deal with both the raw JSON representation and the V1PodSpec instances (depending on where the data comes from). As .as_dict() on the V1PodSpec does not recreate the JSON representation correctly (for example, volume_mounts vs. volumeMounts), we went the other way. If there\u2019s a nicer solution to this, that\u2019d also be appreciated.","issue_id":1659922407700,"origin_id":734124228,"user_origin_id":271710,"create_time":1606375504,"update_time":1606375504,"id":1662226340480,"updated_at":"2022-09-03T17:32:20.479000Z","created_at":"2022-09-03T17:32:20.479000Z"},{"_id":"63138fa493201c528473ca4c","body":"Changing generator is not an easy, especially if you want to be backward compatible.\r\n\r\nDo you consider using `attribute_map`s ? Each model has it. For instance: https:\/\/github.com\/tomplus\/kubernetes_asyncio\/blob\/master\/kubernetes_asyncio\/client\/models\/v1_pod_status.py#L51\r\n\r\nI think, we can add a simple function to module \"utils\" which takes object as input and returns dict using this map. What do you think?","issue_id":1659922407700,"origin_id":737232289,"user_origin_id":1572228,"create_time":1606915967,"update_time":1606916132,"id":1662226340482,"updated_at":"2022-09-03T17:32:20.481000Z","created_at":"2022-09-03T17:32:20.481000Z"},{"_id":"63138fa493201c528473ca4d","body":"That would effectively be a re-implementation of what ``__serialize`` does, which I suppose would be fine.","issue_id":1659922407700,"origin_id":737300267,"user_origin_id":271710,"create_time":1606922655,"update_time":1606922655,"id":1662226340484,"updated_at":"2022-09-03T17:32:20.483000Z","created_at":"2022-09-03T17:32:20.483000Z"}]

As far as I can tell, it does not use any attributes of the ApiClient. The advantage of having it as function would be to have it available without needing...

horazont

Reconnect watcher if server ends empty response

[{"_id":"63138fa493201c528473ca4e","body":"It looks like this behavior was introduced in https:\/\/github.com\/tomplus\/kubernetes_asyncio\/commit\/61852ea42d4ec562d61cbe86d391e5de8a88e75c, as part of https:\/\/github.com\/tomplus\/kubernetes_asyncio\/pull\/22.\r\n\r\n@olitheolix, could you clarify why the asyncio watcher should stop if the server has responded with an empty response? From my perspective, it seems like it'd be better if the watcher remained connected (reconnect) unless the user had specified a timeout. I just want to be sure I'm not missing something...\r\n\r\nI notice that three test cases are failing - I'll address these as soon as I understand the above.","issue_id":1659922407704,"origin_id":593113138,"user_origin_id":144119,"create_time":1583078489,"update_time":1583078489,"id":1662226340836,"updated_at":"2022-09-03T17:32:20.836000Z","created_at":"2022-09-03T17:32:20.836000Z"},{"_id":"63138fa493201c528473ca4f","body":"@JacobHenner From what I can remember, it was about the separation of concerns. The function should only deal with the current event stream and return once K8s closed that stream. The caller can then implement the desired retry\/reconnect\/whatever logic itself. That, at least, is how I did it In my personal version of the `Watcher` :)\r\n\r\nIn other words, no, I do not think you are missing anything here.","issue_id":1659922407704,"origin_id":593265040,"user_origin_id":6856372,"create_time":1583135139,"update_time":1583135139,"id":1662226340839,"updated_at":"2022-09-03T17:32:20.839000Z","created_at":"2022-09-03T17:32:20.839000Z"},{"_id":"63138fa493201c528473ca50","body":"I'd like to keep this client similar to the official client as possible. Are we sure that empty line means that server is disconnected? I don't see such checks in the official client.","issue_id":1659922407704,"origin_id":593398867,"user_origin_id":1572228,"create_time":1583155353,"update_time":1583155353,"id":1662226340842,"updated_at":"2022-09-03T17:32:20.841000Z","created_at":"2022-09-03T17:32:20.841000Z"},{"_id":"63138fa493201c528473ca51","body":"> The function should only deal with the current event stream and return once K8s closed that stream. The caller can then implement the desired retry\/reconnect\/whatever logic itself.\r\n\r\nAh, I see. From my perspective, the library should be responsible for maintaining the connection unless a user-specified timeout has elapsed or the server has responded with some sort of error condition. I've seen other comments that lead me to believe this is the expected behavior from a client - https:\/\/github.com\/kubernetes\/kubernetes\/issues\/6513#issuecomment-90721269, for example.\r\n\r\n\r\n\r\n> Are we sure that empty line means that server is disconnected?\r\n\r\nIt looks like it could mean a few things - https:\/\/github.com\/kubernetes\/kubernetes\/blob\/79e1ad2f4bbd05b1e56b7b57b63b2c1d67b90156\/staging\/src\/k8s.io\/apiserver\/pkg\/endpoints\/handlers\/watch.go#L212-L261\r\n\r\n- Timeout expired\r\n- Marked \"done\"\r\n- Error encountered\r\n\r\n> I don't see such checks in the official client.\r\n\r\nIn the official Python client:\r\n\r\nhttps:\/\/github.com\/kubernetes-client\/python-base\/blob\/d30f1e6fd4e2725aae04fa2f4982a4cfec7c682b\/watch\/watch.py#L141-L157\r\n\r\nWhen iter_resp_lines(resp) returns a value which evaluates to False (e.g. the k8s api returns an empty response), iteration stops. Unless _stop has been set to True, the client will attempt to reconnect.","issue_id":1659922407704,"origin_id":593730261,"user_origin_id":144119,"create_time":1583201669,"update_time":1583201669,"id":1662226340844,"updated_at":"2022-09-03T17:32:20.843000Z","created_at":"2022-09-03T17:32:20.843000Z"},{"_id":"63138fa493201c528473ca52","body":"I got it, the official client \"ignore\" empty lines. In my opinion it's better if a library does mandatory retries to simplify end-client's code.","issue_id":1659922407704,"origin_id":593828558,"user_origin_id":1572228,"create_time":1583224216,"update_time":1583224216,"id":1662226340845,"updated_at":"2022-09-03T17:32:20.845000Z","created_at":"2022-09-03T17:32:20.845000Z"},{"_id":"63138fa493201c528473ca53","body":"> I got it, the official client \"ignore\" empty lines. In my opinion it's better if a library does mandatory retries to simplify end-client's code.\r\n\r\nShould I adjust the tests accordingly?","issue_id":1659922407704,"origin_id":593971423,"user_origin_id":144119,"create_time":1583244861,"update_time":1583244861,"id":1662226340847,"updated_at":"2022-09-03T17:32:20.847000Z","created_at":"2022-09-03T17:32:20.847000Z"},{"_id":"63138fa493201c528473ca54","body":"Yes, It'd be great.","issue_id":1659922407704,"origin_id":594174844,"user_origin_id":1572228,"create_time":1583270434,"update_time":1583270434,"id":1662226340849,"updated_at":"2022-09-03T17:32:20.849000Z","created_at":"2022-09-03T17:32:20.849000Z"},{"_id":"63138fa493201c528473ca55","body":"# [Codecov](https:\/\/codecov.io\/gh\/tomplus\/kubernetes_asyncio\/pull\/96?src=pr&el=h1) Report\n> :exclamation: No coverage uploaded for pull request base (`master@0bdd0f7`). [Click here to learn what that means](https:\/\/docs.codecov.io\/docs\/error-reference#section-missing-base-commit).\n> The diff coverage is `88.88%`.\n\n[](https:\/\/codecov.io\/gh\/tomplus\/kubernetes_asyncio\/pull\/96?src=pr&el=tree)\n\n```diff\n@@ Coverage Diff @@\n## master #96 +\/- ##\n=========================================\n Coverage ? 93.31% \n=========================================\n Files ? 23 \n Lines ? 1585 \n Branches ? 0 \n=========================================\n Hits ? 1479 \n Misses ? 106 \n Partials ? 0 \n```\n\n\n| [Impacted Files](https:\/\/codecov.io\/gh\/tomplus\/kubernetes_asyncio\/pull\/96?src=pr&el=tree) | Coverage \u0394 | |\n|---|---|---|\n| [kubernetes\\_asyncio\/watch\/watch.py](https:\/\/codecov.io\/gh\/tomplus\/kubernetes_asyncio\/pull\/96\/diff?src=pr&el=tree#diff-a3ViZXJuZXRlc19hc3luY2lvL3dhdGNoL3dhdGNoLnB5) | `94.23% <88.88%> (\u00f8)` | |\n| [kubernetes\\_asyncio\/watch\/\\_\\_init\\_\\_.py](https:\/\/codecov.io\/gh\/tomplus\/kubernetes_asyncio\/pull\/96\/diff?src=pr&el=tree#diff-a3ViZXJuZXRlc19hc3luY2lvL3dhdGNoL19faW5pdF9fLnB5) | `100.00% <0.00%> (\u00f8)` | |\n| [kubernetes\\_asyncio\/config\/kube\\_config\\_test.py](https:\/\/codecov.io\/gh\/tomplus\/kubernetes_asyncio\/pull\/96\/diff?src=pr&el=tree#diff-a3ViZXJuZXRlc19hc3luY2lvL2NvbmZpZy9rdWJlX2NvbmZpZ190ZXN0LnB5) | `94.25% <0.00%> (\u00f8)` | |\n| [kubernetes\\_asyncio\/config\/incluster\\_config.py](https:\/\/codecov.io\/gh\/tomplus\/kubernetes_asyncio\/pull\/96\/diff?src=pr&el=tree#diff-a3ViZXJuZXRlc19hc3luY2lvL2NvbmZpZy9pbmNsdXN0ZXJfY29uZmlnLnB5) | `85.10% <0.00%> (\u00f8)` | |\n| [kubernetes\\_asyncio\/config\/exec\\_provider\\_test.py](https:\/\/codecov.io\/gh\/tomplus\/kubernetes_asyncio\/pull\/96\/diff?src=pr&el=tree#diff-a3ViZXJuZXRlc19hc3luY2lvL2NvbmZpZy9leGVjX3Byb3ZpZGVyX3Rlc3QucHk=) | `100.00% <0.00%> (\u00f8)` | |\n| [kubernetes\\_asyncio\/config\/google\\_auth.py](https:\/\/codecov.io\/gh\/tomplus\/kubernetes_asyncio\/pull\/96\/diff?src=pr&el=tree#diff-a3ViZXJuZXRlc19hc3luY2lvL2NvbmZpZy9nb29nbGVfYXV0aC5weQ==) | `100.00% <0.00%> (\u00f8)` | |\n| [kubernetes\\_asyncio\/config\/config\\_exception.py](https:\/\/codecov.io\/gh\/tomplus\/kubernetes_asyncio\/pull\/96\/diff?src=pr&el=tree#diff-a3ViZXJuZXRlc19hc3luY2lvL2NvbmZpZy9jb25maWdfZXhjZXB0aW9uLnB5) | `100.00% <0.00%> (\u00f8)` | |\n| [kubernetes\\_asyncio\/config\/kube\\_config.py](https:\/\/codecov.io\/gh\/tomplus\/kubernetes_asyncio\/pull\/96\/diff?src=pr&el=tree#diff-a3ViZXJuZXRlc19hc3luY2lvL2NvbmZpZy9rdWJlX2NvbmZpZy5weQ==) | `93.78% <0.00%> (\u00f8)` | |\n| [kubernetes\\_asyncio\/utils\/\\_\\_init\\_\\_.py](https:\/\/codecov.io\/gh\/tomplus\/kubernetes_asyncio\/pull\/96\/diff?src=pr&el=tree#diff-a3ViZXJuZXRlc19hc3luY2lvL3V0aWxzL19faW5pdF9fLnB5) | `100.00% <0.00%> (\u00f8)` | |\n| [kubernetes\\_asyncio\/config\/\\_\\_init\\_\\_.py](https:\/\/codecov.io\/gh\/tomplus\/kubernetes_asyncio\/pull\/96\/diff?src=pr&el=tree#diff-a3ViZXJuZXRlc19hc3luY2lvL2NvbmZpZy9fX2luaXRfXy5weQ==) | `100.00% <0.00%> (\u00f8)` | |\n| ... and [14 more](https:\/\/codecov.io\/gh\/tomplus\/kubernetes_asyncio\/pull\/96\/diff?src=pr&el=tree-more) | |\n\n------\n\n[Continue to review full report at Codecov](https:\/\/codecov.io\/gh\/tomplus\/kubernetes_asyncio\/pull\/96?src=pr&el=continue).\n> **Legend** - [Click here to learn more](https:\/\/docs.codecov.io\/docs\/codecov-delta)\n> `\u0394 = absolute <relative> (impact)`, `\u00f8 = not affected`, `? = missing data`\n> Powered by [Codecov](https:\/\/codecov.io\/gh\/tomplus\/kubernetes_asyncio\/pull\/96?src=pr&el=footer). Last update [0bdd0f7...ff5894a](https:\/\/codecov.io\/gh\/tomplus\/kubernetes_asyncio\/pull\/96?src=pr&el=lastupdated). Read the [comment docs](https:\/\/docs.codecov.io\/docs\/pull-request-comments).\n","issue_id":1659922407704,"origin_id":609114327,"user_origin_id":8655789,"create_time":1586050424,"update_time":1586050664,"id":1662226340853,"updated_at":"2022-09-03T17:32:20.852000Z","created_at":"2022-09-03T17:32:20.852000Z"},{"_id":"63138fa493201c528473ca56","body":"Is there any progress on this PR? ","issue_id":1659922407704,"origin_id":668373234,"user_origin_id":8097526,"create_time":1596515505,"update_time":1596515505,"id":1662226340856,"updated_at":"2022-09-03T17:32:20.855000Z","created_at":"2022-09-03T17:32:20.855000Z"}]

If the server sends an empty response (e.g. a server-side timeout was exceeded), the watcher should reconnect unless the user has specified a timeout. This is similar to the behavior...

JacobHenner

watch stream set timeout_seconds=0 not work

[]

as example code: ``` async def watch_job_event(namespace: str, callback: asyncio.coroutines): """ watch job event""" async with ApiClient() as api: batchv1 = client.BatchV1Api(api) w = watch.Watch() logging.info("begin to watch job events")...

flyhighzy