[kubernetes] Add ability to customize worker/scheduler pod names

[bolliger32]

In dask-kubernetes, you can use the kubernetes.worker_name value to add each user's name to the worker pod names. Is that customization something that can be done in Dask-gateway? I haven't figured that out yet - if it's not possible, I think that would be a useful feature to add. Would be nice for more quickly filtering pods and diagnosing any issues a particular user may be having. Happy to work on that but wanted to check first and see that I wasn't just missing some existing functionality somewhere

[droctothorpe]

https://gateway.dask.org/api-server.html#c.KubeClusterConfig.scheduler_extra_pod_labels

https://gateway.dask.org/api-server.html#c.KubeClusterConfig.worker_extra_pod_labels

We use cluster_options to inject usernames at spawn time which are consumed downstream for cost calculation by Kubecost and are leveraged for other kinds of administrative functions like you mentioned.

[jcrist]

Is that customization something that can be done in Dask-gateway? I haven't figured that out yet - if it's not possible, I think that would be a useful feature to add.

This is currently not possible, but could be supported. I'd advise storing this in an annotation or label instead though (annotations have more flexibility). There are restrictions on pod names (length and supported characters) that mean any username would have to be quoted before templating in, and the total length trimmed. Annotations are much less restricted.

As @droctothorpe pointed out, you could use cluster_options and a handler to inject labels/annotations with the username included (docs here: https://gateway.dask.org/cluster-options.html). We could add a user annotation by default if needed, but I'd prefer to leave it out if possible.

[bolliger32]

Thanks @jcrist and @droctothorpe. We are indeed using cluster_options to inject the username for other purposes, however there's one use case where I don't think this gets us to where we want to be. We've been using the GCP console GUI (the one at console.cloud.google.com) to monitor various user's pods. There it's very easy to filter by pod name but not (at least as far as I can tell) to filter by label/annotation. If either of you (or anyone) knows an easy way to filter pods by a label in the GCP console, let me know!

[droctothorpe]

I haven't used the GCP console but can you use kubectl from within it?

If so, kubectl get pod -l <key=value> should get you what you want.

https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#api

[bolliger32]

You can't directly use kubectl to filter workloads via the console, as far as I know.

[cslovell]

I know I'm late to the game - @droctothorpe I'm curious if you have a code snippet on how this is done. Also aiming to achieve some insight into per-user costs.

[cslovell]

Nevermind - seems the following works in the Helm chart values file:

dask-gateway: 
  gateway: 
    backend: ... [omitted for brevity]
    extraConfig:
      optionHandler: |
        from dask_gateway_server.options import Options, Integer, Float, String
        def cluster_options(user):  
            def option_handler(options):
                if ":" not in options.image:
                    raise ValueError("When specifying an image you must also provide a tag")
                extra_labels = {
                    "hub.jupyter.org/username": user.name
                }
                return {
                    "worker_cores": options.worker_cores,
                    "worker_memory": int(options.worker_memory * 2 ** 30),
                    "image": options.image,
                    "scheduler_extra_pod_labels": extra_labels,
                    "worker_extra_pod_labels": extra_labels,
                }
            return Options(
                Float("worker_cores", default=0.8, min=0.8, max=4.0, label="Worker Cores"),
                Float("worker_memory", default=3.3, min=1, max=8, label="Worker Memory (GiB)"),
                String("image", default="pangeo/base-notebook:2021.05.04", label="Image"),
                handler=option_handler,
            )
        c.Backend.cluster_options = cluster_options