aiohttp: Response payload is not completed

[hrybun]

After about 3 minutes I got an error: aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed

Run:

async def get_events(v1, field_selector: str):
    w = watch.Watch()
    async with w.stream(
            v1.list_event_for_all_namespaces,
            field_selector=field_selector,

    ) as stream:
        return stream

async def show_events(events):
    async for event in events:
        evt, obj = event["type"], event["object"]
        print(
            "-" * 40,
            f"\nMETHOD: {evt}\n"
            f"KIND: {obj.kind}\n"
            f"NAME: {obj.metadata.name}\n"
            f"NAMESPACE: {obj.metadata.namespace}\n"
            f"MESSAGE: {obj.message}\n",
        )

async def watch_events() -> None:
    async with ApiClient() as api:
        v1 = client.CoreV1Api(api)
        res_events = await asyncio.gather(
            *[
                get_events(v1, f"involvedObject.kind={name}")
                for name in ["Cluster", "Node"]
            ]
        )
        await asyncio.gather(*[show_events(event) for event in res_events])

python: 3.7.2 and 3.9.12 kubernetes_asyncio: 18.20.0 and 24.2.2

[tomplus]

Thanks for reporting the issue. Could you send information about your cluster - K8s version, cloud provider?

[hrybun]

Now using k8s version 1.19 soon will add newer versions. We are developing cloud, now it is in closed beta testing.

[vishnukv-facets]

I am also facing this issue, will this be fixed?

[tomplus]

Unfortunately I'm not able to recreate the issue. @vishnukv-facets could you share more details about your case - your cluster type, k8s version, what kind of interfaces you are watching etc. Thanks.

[vishnukv-facets]

I am using a 1.24.9 AKS cluster in azure , python - 3.9, kubernetes_asyncio latest pip package

async with watch.Watch().stream(v1.list_namespaced_secret, namespace="default") as stream:
  async for event in stream:

The above piece of code always throws the error even if i add timeout_seconds it still throws the aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed error

sometimes i get this error too

EDIT: ~~I am only getting these errors inside the container and not when running on my laptop~~ After running for sometime i also got the same error in my laptop as well ( python 3.9 , M1 MacOS )

[EricKolibacz]

Had the same problem:

All Job's above 60 seconds failed in a watch coroutine

async with kubernetes_asyncio.watch.Watch() as w:
    async for event in w.stream(...)
        ...

with Response payload is not completed

Our setup: kubernetes v1.23.0 with an additional ha-proxy.

I found this issue in the aiohttp GitHub repo. One of the answers made me believe that we are facing the same issue with our kubernetes setup. The ha-proxy has its own timeout set (for us at 50 seconds). So all our jobs taking more than 50 seconds receive the error. (this is due to the fact that the proxy kills all requests which do not send any data for some time and since the watch coroutine can idle if no pod event is happening the proxy closes the connection).

Fixed it by using a while loop and periodically requesting the status of the pod. Not as elegant as using watch but it fixed the issue :tada:.

Alternatively, you probably could increase the timeout of the proxy (in the case for the ha proxy, configure it in the /etc/haproxy/haproxy.cfg file).