How can I disable the port-forward?

[jacobtomlinson]

How can I disable this port-forward? I do not need it because dask scheduler/dashboard is accessible through the kubernetes service. It is not necessary to run "kubectl port-forward", it should not be used.

Originally posted by @project-administrator in https://github.com/dask/dask-kubernetes/issues/489#issuecomment-1177473955

[jacobtomlinson]

It is my understanding that the Kubernetes service can act as a layer 7 proxy, so web services can be accessed through it. I didn't think this is often used though. Also Dask uses a custom TCP protocol for communication which I don't think the Kubernetes service can proxy. If my assumption is wrong here I'd be keen to learn more though.

The KubeCluster object needs to open a connection with the scheduler. If the scheduler has been exposed directly with a load balancer or a node port then it will connect that way. If it is using a ClusterIP then it will attempt to connect to that IP directly (which will work if the KubeCluster object is within a python process in the cluster) or if that fails the last resort is to use port-forward.

https://github.com/dask/dask-kubernetes/blob/e83e22f032f0f0e990a04589ae5246e7eee63560/dask_kubernetes/common/networking.py#L16-L48

We could definitely add another option here that avoids the port forward, but it would require the user to supply the address that the KubeCluster can use to connect to the scheduler.

[project-administrator]

No, Kubernetes service is just a TCP-proxy basically.

KubeCluster object should be able to access the scheduler through the kubernetes service resource by connecting to the host address which matches the kubernetes service resource name.

The connection should succeed once the pods to which kubernetes service points are ready.

Kubernetes service will balance traffic between all pods where it points to (i.e. forward your connection to a random pod if it points to more than one pod).

So, if dask should run only one dashboard instance (pod), then it should have a separate kubernetes service which points only to the dashboard.

[jacobtomlinson]

Thanks for the info. Do you have any good resources where I can learn more? I haven't seen anyone using the Kubernetes service in practice before. And googling "Kubernetes Service" gives lots of unrelated results.

So in terms of adding support for this is there a way we can discover the address of the service and check that it is accessible? Or is this something you would expect to provide as a user?

[project-administrator]

https://kubernetes.io/docs/concepts/services-networking/service/

I can see that KubeCluster class creates three resources on the Kubernetes cluster: pods, service, podDisruptionBudget. So, we already have the Kubernetes service created.

In order to connect to the dashboard (from KubeCluster or with a dask client class), just use the Kubernetes service name as the DNS hostname to connect to. So, you do not need to modify code to add any discovery mechanism. This is already covered by the Kubernetes service.

Another question is that cluster.scale() creates more pods which might be added to the same Kubernetes service. If dask dashboard runs on all of these scaled dask pods, then I guess we are ok. If dask dashboard is something that should be running only on one pod (i.e. it's designed to be running only one instance of the dashboard), then KubeCluster should also create and use a separate Kubernetes service which is used only for the connection to the dask dashboard.

[mmccarty]

Thoughts on an opt-out flag for port-forwarding?

[jacobtomlinson]

In order to connect to the dashboard (from KubeCluster or with a dask client class), just use the Kubernetes service name as the DNS hostname to connect to. So, you do not need to modify code to add any discovery mechanism. This is already covered by the Kubernetes service.

I am a little confused. I thought you meant accessing via the proxy in the Kubernetes API server, but you mean accessing the dask-scheduler service directly right? We already support this. We attempt to connect to the service directly and only port-forward if that fails.

https://github.com/dask/dask-kubernetes/blob/e83e22f032f0f0e990a04589ae5246e7eee63560/dask_kubernetes/common/networking.py#L39

So if you're seeing port-forwarding happening when you don't expect it that may be a bug.

Another question is ...

Would you mind opening another issue to ask a different question so that we can keep this thread on topic.

Thoughts on an opt-out flag for port-forwarding?

I'm not a fan. In order to start the cluster we need to connect to the scheduler. We only port forward when all else fails, and if we allow folks to disable the port forwarding then cluster creation will just fail. I'm also keen to avoid kwarg creep.

[project-administrator]

OK, I'm not sure why does it fail the _is_service_available check and falls back to port-forwarding. I am able to connect to dask by using the Kubernetes service after KubeCluster is created. Since I've verified that I'm able to use the Kubernetes service for the connection to dask, I see no reason to use the port-forwarding. It fails the socket.getaddrinfo(host, port) check, but still I'm able to connect to dask via the Kubernetes service from the same pod where I run the code calling the KubeCluster.

[jacobtomlinson]

Are you able to open another Python process and see why socket.getaddrinfo(host, port) is failing?

[jacobtomlinson]

I'm going to close this out due to the inactivity. Feel free to comment here if other folks run into this.