distributed

distributed copied to clipboard

LocalCluster fails to spawn requested number of workers (capped at 5 workers on Windows)

3

### Description When creating a `LocalCluster` with `n_workers` > 5, the cluster consistently only spawns exactly 5 workers, regardless of the requested number. This appears to be a hard limit...

jianlinshi

documentation not updated following #3029

3

Following closure of https://github.com/dask/distributed/issues/1389 after merge allowing work stealing for resources-restricted tasks, it seems doc has not been updated. https://distributed.dask.org/en/stable/work-stealing.html#restrictions still states: "If a task has been specifically restricted to...

vianneylanguille

Add MultiprocessingAuthkeyPlugin to propagate authkey to Dask workers

6

Closes #9122 - [ ] Tests added / passed - [x] Passes `pre-commit run --all-files`

moi90

Add ForwardLoggingPlugin and ForwardOutput to built-in worker plugins documentation

2

`ForwardLoggingPlugin` and `ForwardOutput` were missing from the "Built-In Worker Plugins" section. This PR adds them. - [ ] Tests added / passed - [x] Passes `pre-commit run --all-files`

moi90

Local Python env Cluster

10

- [x] Closes https://github.com/dask/dask-labextension/issues/82 - [ ] Tests added / passed - [x] Passes `black distributed` / `flake8 distributed` / `isort distributed` Add a new distributed cluster manager that provides...

DPeterK

Forward `multiprocessing.current_process().authkey` to workers

3

I'm using a [`multiprocessing` Manager](https://docs.python.org/3/library/multiprocessing.html#managers) to enable communication between the main process and it's workers (for progress reporting). I also want to support `dask_jobqueue.SLURMCluster` as a method to distribute work....

moi90

improve support for grouped workers in `SpecCluster`

5

This PR supersedes #9104 and implements missing support for grouped workers in `SpecCluster`. As described in #9104, support for grouped workers was originally implemented in #3013 and is still documented...

alisterburt

Check plugin registration status

1

Closes #9106 - [ ] Tests added / passed - [x] Passes `pre-commit run --all-files` Adds a client side function to check if a plugin is registered on either the...

jayavenkatesh19

Allow min worker queue length of 0

3

My use case for Dask is primarily for orchestrating the execution of long running functions. By long running, I mean minutes, and sometimes hours. Tasks can be submitted to a...

super-ibby

check if a Plugin is present

2

We have `register_plugin` and `unregister_*_plugin` I would like a function to check if a plugin is registered. I guess technically this is a bit race conditiony, but it would be...

oxinabox