Florian Jetter

Worker can memory overflow by fetching too much data at once

12

There is a possibility for a worker to suicide by fetching too much data at once Old version: https://github.com/dask/distributed/blob/84cbb099e0d0f1e1947d51d957f0b65e9b0635dd/distributed/worker.py#L2754-L2757 New version: https://github.com/dask/distributed/blob/3647cfefd004281dac0ffa92349e23bc2a94d68a/distributed/worker_state_machine.py#L1425-L1429 Simple example Let's assume that keys have on...

Increase timeouts in test_asyncprocess::test_simple

7

This test flaked over in https://github.com/dask/distributed/pull/6991 The functionality appeared to be OK but the process join took ~6 seconds which tripped the `assert dt

Simplify usage of Queues in nanny

3

This is a cleanup of the nanny around the usage of multiprocessing queues. It ensures that queues are only closed once and no exceptions are swallowed

There is a race condition in `SpecCluster.close` that can lock potentially indefinitely if a cluster is closing while instances are spawned at the same time. It is not clear, yet,...

Simplify decide_worker

5

This came up during review of https://github.com/dask/distributed/pull/6614#discussion_r958287053 Bottom line is that this code path is only there for performance optimization and it approximates the decision performed by decide worker (it...

Reuse of keys in blockwise fusion can cause spurious KeyErrors on distributed cluster

16

Subsequent Blockwise layers are currently fused into a single layer. This reduces the number of tasks, the overhead and is very generally a good thing to do. Currently, the fused...

Mamba and conda solve of CI environment files installs ancient s3fs version causing poor performance

9

When installing our test-environments that we are using in CI I am pulling in an ancient version of `s3fs` that is causing severe performance problems, see https://github.com/dask/dask/issues/10275 and https://github.com/fsspec/filesystem_spec/issues/1261 It...

Ensure `map_partitions` returns `Series` object if function returns scalar

3

- [x] Closes https://github.com/dask-contrib/dask-expr/issues/625 - [x] Tests added / passed - [x] Passes `pre-commit run --all-files`

[WIP] Pass non HLG objects wout materialization

2

Sibling and description https://github.com/dask/distributed/pull/7942

Document that describe is using approximate algorithms

6

Calculating quantiles is difficult and expensive. For operations like `{DataFrame|Series}.describe` or `quantile` we're actually automatically falling back to an approximate algorithm. This can be confusing, particularly for new users who...