dask operator scheduler service name

[webcoderz]

is it possible to change the service name having issues trying to connect my workers and other containers to the scheduler using dask-scheduler:8786 ? the dask-scheduler:8786 value is hardcoded in another image container so trying to avoid changing that elsewhere

[webcoderz]

apiVersion: v1
kind: Service
metadata:
  name: dask-scheduler
spec:
  ports:
    - name: "8786"
      port: 8786
      targetPort: 8786
    - name: "8787"
      port: 8787
      targetPort: 8787
  selector:
      dask.org/cluster-name: dask
      dask.org/component: scheduler

made my own service but this is a hacky workaround..

[jacobtomlinson]

~~Could you share how you are creating the cluster?~~

Just spotted in https://github.com/dask/helm-chart/issues/323 that you're using the operator.

The scheduler service name is hard-coded to be the cluster name with -scheduler on the end.

https://github.com/dask/dask-kubernetes/blob/9f115a4e686f4efd8f00d83aec9522889c242aa1/dask_kubernetes/operator/operator.py#L52

We do this to ensure the service name is unique. If you have a hard-coded service of dask-scheduler in your container image and you wanted to override the service name to match then you would only be able to deploy one Dask cluster on your Kubernetes cluster at a time or you will get name conflicts.

The scheduler address is injected into the worker pods at runtime via the DASK_SCHEDULER_ADDRESS environment variable.

https://github.com/dask/dask-kubernetes/blob/9f115a4e686f4efd8f00d83aec9522889c242aa1/dask_kubernetes/operator/operator.py#L83-L84

If you omit the address in the dask-worker command then Dask will pick up this address automatically, this is what we do by default. If you need to hard code dask-scheduler:8786 in your custom image for other reasons I recommend you update your image to set it to ${DASK_SCHEDULER_ADDRESS:=dask-scheduler:8786} so the the environment variable still overrides the address correctly.