Can not connect to k8s websocket deployed in Rancher

[dbalabka]

Describe the issue: Deploy k8s cluster in a rancher. Try to connect with default settings. You will see:

2024-01-02 21:19:05,453 - asyncio - ERROR - Task exception was never retrieved
future: <Task finished name='Task-26' coro=<PortForward._sync_sockets() done, defined at /home/torinaki/src/demand-forecasting/.venv/lib/python3.10/site-packages/kr8s/_portforward.py:168> exception=AssertionError('Only absolute URLs without path part are supported')>
Traceback (most recent call last):
  File ".../.venv/lib/python3.10/site-packages/kr8s/_portforward.py", line 171, in _sync_sockets
    async with self._connect_websocket() as ws:
  File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File ".../.venv/lib/python3.10/site-packages/kr8s/_portforward.py", line 150, in _connect_websocket
    async with self.pod.api.open_websocket(
  File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File ".../.venv/lib/python3.10/site-packages/kr8s/_api.py", line 184, in open_websocket
    async with aiohttp.ClientSession(
  File ".../.venv/lib/python3.10/site-packages/aiohttp/client.py", line 261, in __init__
    self._base_url.origin() == self._base_url
AssertionError: Only absolute URLs without path part are supported

because Rancher provides the following URL to connect to websockets: https://rancher.example.com/k8s/clusters/c-m-4wx4gzpx

Minimal Complete Verifiable Example:

TBD

Anything else we need to know?:

Environment:

• Dask version: 2023.11.0
• Python version: 3.10
• Operating System: Linux
• Install method (conda, pip, source): poetry

[dbalabka]

To fix the issue, aiohttp needs to support the path in base_url: https://github.com/aio-libs/aiohttp/issues/6647

It works for me in my patched version.

[jacobtomlinson]

Would you mind opening an issue upstream in kr8s for this? Preferably with a small example we can use to reproduce the issue (I know we would need Rancher but the snippet that reproduces on your end would still be useful).

[dbalabka]

@jacobtomlinson , I think the simplest way to reproduce the problem is to put k8s API behind the proxy and add an extra path (proxy http://127.0.0.1/somepath to http://127.0.0.1) and adjust kubectl settings. The Nginx reverse proxy_pass rule can help:

location /somepath {
  proxy_pass http://127.0.0.1
}

[jacobtomlinson]

It looks like kubectl can handle this for you.

$ kubectl proxy --api-prefix=/somepath/
Starting to serve on 127.0.0.1:8001

$ curl localhost:8001/somepath/version
{
  "major": "1",
  "minor": "29",
  "gitVersion": "v1.29.0",
  "gitCommit": "3f7a50f38688eb332e2a1b013678c6435d539ae6",
  "gitTreeState": "clean",
  "buildDate": "2023-12-14T19:18:17Z",
  "goVersion": "go1.21.5",
  "compiler": "gc",
  "platform": "linux/amd64"
}

[jacobtomlinson]

While digging into this I've noticed some other strange behaviour when the Kubernetes API is proxied with an extra path. I opened https://github.com/kr8s-org/kr8s/issues/284 to track.