dask-kubernetes
dask-kubernetes copied to clipboard
dask
Native Kubernetes integration for Dask
When creating a `DaskCluster` with `dask_kubernetes.experimental.KubeCluster` the name you specify gets `-cluster` added onto the name. ```python from dask_kubernetes.experimental import KubeCluster cluster = KubeCluster(name='foo') ``` ```console $ kubectl get dsk...
The first iteration of the experimental `KubeCluster` stored the default values in the class kwarg defaults and the only way to override them via kwargs. For example `n_workers`: https://github.com/dask/dask-kubernetes/blob/e83e22f032f0f0e990a04589ae5246e7eee63560/dask_kubernetes/experimental/kubecluster.py#L127 https://github.com/dask/dask-kubernetes/blob/e83e22f032f0f0e990a04589ae5246e7eee63560/dask_kubernetes/experimental/kubecluster.py#L138...
The current implementation of `KubeCluster` insists users specify a pod spec for their workers and schedulers and it provides some helper functions to do this and does some heavy customisation...
If a `DaskCluster` is configured with a `NodePort` service but the ports are out of range the `DaskCluster` will be created but the controller logs will error repeatedly. ``` HTTP...
How can I disable this port-forward? I do not need it because dask scheduler/dashboard is accessible through the kubernetes service. It is not necessary to run "kubectl port-forward", it should...
When connecting to my dask cluster using ```python cluster = HelmCluster(release_name='foo') ``` I kept getting the following error: `ConnectionError: kubectl port forward failed`. After debugging a bit, it turned out...
As part of the `dask` backend plugin work for [flyte](https://docs.flyte.org/en/latest/) that I am doing, I have also started creating a golang client for the operator in `dask-kubernetes`. There is an...
Closes #440 This PR adds autoscaling to the Operator by creating a new DaskAutoscaler CRD and then using a `kopf` timer to track it.
This PR is the result of a chat with @bstadlbauer about extending the functionality of the operator. The specific context is what if the user wants the operator to create...