Florian Jetter

How was that error propagated to the client before? `Cluster.scale` typically works asynchronously in the background and does not block so it is difficult to raise exceptions.

So, the code snippet you are posting shows what I would expect to happen. I'm surprised to hear that this worked differently before

> I think it propagated through the cluster.sync(cluster._correct_state) call. That makes sense. What exactly is your expectation? The public API did not change so I assume you are internally using...

> To raise a RuntimeError. With current main it does not. I would like us to find a way for public API to make sense for you. I wouldn't even...

> OK, or even just a public blessed way of waiting for a scale up to complete and reporting any errors that occurred. I believe the "wait for scaleup" is...

This has been introduced by https://github.com/dask/dask/pull/11736 This change was supposed to improve client to scheduler submission time but I noticed that for some workloads it had the opposite effects, particularly...

I haven't tried the version with da.random (which has issues), the issue with flox enabled is quite snappy for me. Even with flox disabled, the changes in #11736 *did* make...

Ok, I also did some measurements and believe this is https://github.com/dask/dask/issues/11735 With https://github.com/dask/dask/pull/11736 we are saving on transmission cost but that's an unrelated thing and it isn't released, yet

> This gets even more complicated when da.store is involved. FYI I intend to change that in https://github.com/dask/dask/pull/11844 which would preserve the array type and avoid many of those problems

Also note that we're working on a new array backend which will change how optimization will be performed. The new backend will also be using what we call an Expression,...