Support passing ownerReference when creating a k8s based Dask cluster - for automatic cleanup of the k8s DaskCluster resources

[consideRatio]

When a user pod has spawned a dask-gateway cluster within the same k8s cluster, one could in theory make that cluster clean itself up by adding a ownerReference pointing to the jupyterhub user server represented as a k8s pod.

I think this must be implemented by having the dask-gateway client to be configured to pass the k8s Pod's unique identifier to the dask-gateway server via the REST api as part of some optional data field. The reason for having it optional, is that only the Kubernetes implementation on how to create a dask-gateway cluster will be able to understand how to use this k8s Pod's identifier to set to ownerReference field of the k8s DaskCluster resource.

Another part to solve is to make sure there is a reliable way for the k8s Pod to be aware of its owns reference. I guess that can be set to some environment variable though by fixed configuration of the k8s Pod's manifest.

In detail, I think the following command needs to be passed information via the options argument.

https://github.com/dask/dask-gateway/blob/b86e3066839c44801c11a51564091f59b705f342/dask-gateway-server/dask_gateway_server/backends/kubernetes/backend.py#L503

Issue action points

• [ ] For @consideRatio to investigate the idea further and provide more practical details on a suggested implementation strategy
• [ ] For anyone to review the feature proposal idea so far

[consideRatio]

This is a strategy I think that can be used to acquire the k8s UID of the pod. Practically, one could then also name the env in a way to configure the dask-gateway client to be configured to pass that information via the dask configuration system that reads ENV.

What remains then would be to expose configuration to reference, and ensure it can be passed from the dask-gateway client to the dask-gateway server, and have the k8s backend be able to handle that information in order to set the ownerReferences field of the DaskCluster manifest it will generate.

apiVersion: v1
kind: Pod
metadata:
  name: test-set-pods-uid-as-env
spec:
  containers:
    - name: test-container
      image: k8s.gcr.io/busybox:1.24
      command: [ "sh", "-c"]
      args:
      - while true; do
          echo -en '\n';
          printenv MY_UID;
          sleep 10;
        done;
      env:
        - name: MY_UID
          valueFrom:
            fieldRef:
              fieldPath: metadata.uid

[consideRatio]

This is a duplicate of #260.