dask
CI: Distributed fixtures not compliant with dask-jobqueue
I hesitated to put this issue as a comment in #567, but I'm under the impression we have two separate problems.
In #562, I override the cleanup fixture from distributed to prevent it from removing the default MainThread loop (so #567 related), but also to make it reload the dask-jobqueue configuration into dask.config. Since distributed last 2022.08.0 version, the loop fixture we used into our tests has also been modified (https://github.com/dask/distributed/commit/172e37f41f84f117ec79a1186a951be6f66db7f3). In this new version, config is also reset, so I directly modify the method distributed.utils_test.reset_config to also reload dask-jobqueue config.
Question is: how could we use these fixtures without these hacks? How to make dask-jobqueue config reset at the same time as the distributed one is reset? Does this need a fix in distributed?
Going further, I have another question of understanding (also expressed here): why do we need the loop fixture? I don't see other Dask cluster deploying system using it. Is this still necessary with SpecCluster?
cc @graingert @jacobtomlinson.
