Florian Jetter

FYI I had an incredibly great experience with pre-shuffle compression for high column cardinality problems, see https://github.com/dask/dask/pull/6259 I never had the time to finish this but the code is straight...

This is a work stealing problem. If we disable work stealing, it works as expected ```python import time import dask import distributed with dask.config.set({"distributed.scheduler.work-stealing": False}): client = distributed.Client(n_workers=4, threads_per_worker=1) root...

I looked into this briefly today and could narrow this down to a couple of issues - Initially, before the execution time could be learned, [unknown-task-duration](https://github.com/dask/distributed/blob/99a2db1cb75e4edf3c9e1c4ed8c691a0a47e71c5/distributed/distributed.yaml#L29) is used for the...

The initial imbalance is cause by work stealing being selecting potential victims greedily in case there are no saturated workers around. Specifically the following lines https://github.com/dask/distributed/blob/99a2db1cb75e4edf3c9e1c4ed8c691a0a47e71c5/distributed/stealing.py#L424-L429 are responsible for quite...

I investigated what's causing the initial spike of stealing events. Very early in the computation we see that slightly less than 7.5k stealing decisions are enacted which causes this initial...

I'm also a big fan of py-spy and would love to see an integration. I guess the way dask and ray approaches profiling is slightly different. In dask we actually...

The "what fixtures should be deprecated" conversation probably belongs into https://github.com/dask/distributed/issues/6806 I suggest to get started on that one before introducing any deprecations really. We can define the fixtures we...

https://github.com/dask/distributed/issues/6869 is an example why I am concerned about letting multiple coroutines for the same task running, i.e. gather_dep and execute. This can cause overlap and our event handlers need...

> I still want to go on with the removal of the 'long-running' state though and I'd like to open a PR just for it. @fjetter you were against that...

Thanks @gjoseph92 ! I'll have a look asap. I suggest to keep the refactoring split off in the commit for now and we can decide if this is rather a...