Introduce public-facing `distributed.testing` module and deprecate `distributed.utils_test`

[hendrikmakait]

In #6806, we have reached support for the idea of introducing a public-facing distributed.testing module that contains all the testing utilities that we would like friends-of-dask and users to use in their CI. distributed.utils_test should then be deprecated in favor of distributed.testing issuing a deprecation warning pointing toward distributed.testing whenever someone imports distributed.utils_test. All imports from utils_test must still work after the restructuring. Whenever possible, we should add a more pointed deprecation warning to utilities that points toward a preferred alternative.

The contents of distributed.utils_test should be moved into a private module, e.g. distributed._testing. From there, we import a subset of the testing utilities into distributed.testing to be exposed as public-facing utilities. These utilities should be well-documented and represent the best practices we follow for testing distributed that we encourage others to use in CI. We want to highlight that these utilities are not considered stable with clear deprecation cycles, but

when we make a change:

• it will it will probably cause your test suite to fail
• but any failure we cause should be actionable
• and we should come help you fix it
• and most importantly you should always be able to copy paste an old version to unblock any PRs and issues with higher priority

(https://github.com/dask/distributed/issues/6806#issuecomment-1205477913)

and may be changed by us to enforce best practices, but set the expectation that we will support the Dask ecosystem in adapting to these changes through documentation or active development. The private contents of distributed.utils_test should only be used within distributed and can therefore be changed without affecting anybody in the Dask ecosystem.

When it comes to the concrete contents of distributed.testing, we have the following guidelines:

• Do not export fixtures, use context managers instead.
• Less is more.
• Document properly.

After a first search of friends-of-dask, I have found the following utilities in use:

• loop (8x)
• cluster (5x)
• gen_cluster (5x)
• loop_in_thread (4x)
• cleanup (3x)
• popen (2x)
• captured_logger (2x)
• inc (2x)
• client (1x)
• cluster_fixture (1x)
• gen_test (1x)
• raises_with_cause (1x)
• slowinc (1x)
• varying (1x)
• import utils_test (1x)

Consequently, I think loop, cluster and gen_cluster may be good candidates to start with in distributed.testing. For the remainder, we should take a look at the tests in question to see if we should rewrite the test, copy+paste the utility, or include it in distributed.testing.