dask-kubernetes-operator-role-cluster clusterrole does not have the needed ACL against pods/portforward resource

[oe-hbk]

Describe the issue: The dask-kubernetes-operator pod shows an 403 Forbidden error when trying to access the k8s api. It does not seem to have the right cluster role permissions

[2024-10-08 21:48:24,704] httpx                [INFO    ] HTTP Request: GET https://10.233.0.1/api/v1/namespaces/MYNAMESPACE/pods/MYPOD/portforward?name=MYPOD&namespace=MYNAMESPACE&ports=80&_preload_content=false " HTTP/1.1 403 Forbidden"

Execcing into the pod and trying the same call against the API.

kubectl exec -it -n dask-system dask-kubernetes-operator-78d4b784cf-4r455 -- sh

$ SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount
$ NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace)
$ TOKEN=$(cat ${SERVICEACCOUNT}/token)
$ CACERT=${SERVICEACCOUNT}/ca.crt
$ curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET 'https://10.233.0.1/api/v1/namespaces/MYNAMESPACE/pods/MYPOD/portforward?name=MYPOD&namespace=MYNAMESPACE&ports=80&_preload_content=false'
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "pods \"MYPOD\" is forbidden: User \"system:serviceaccount:dask-system:dask-kubernetes-operator
\" cannot get resource \"pods/portforward\" in API group \"\" in the namespace \"MYNAMESPACE\"",
  "reason": "Forbidden",
  "details": {
    "name": "MYPOD",
    "kind": "pods"
  },
  "code": 403
}$

Editing the clusterrole,

$ kubectl edit clusterrole -n dask-system dask-kubernetes-operator-role-cluster

And adding pods/portforward

Around https://github.com/dask/dask-kubernetes/blob/ab1be696d03a8963f0db120e0de993f3eda12930/dask_kubernetes/operator/deployment/helm/dask-kubernetes-operator/templates/clusterrole.yaml#L34

and restarting the application pod corrected the problem.

Environment:

• Dask version: dask-kubernetes-operator-2024.5.0
• Python version:
• Operating System: Rocky 8
• Install method (conda, pip, source): helm chart