make_current/clear_current are deprecated in Tornado

[QuLogic]

What happened: make_current and clear_current appear to be deprecated in Tornado 6.2, which causes errors in tests, since warnings are treated as errors. Though it's my mistake that I left that on, it's probably something that should be fixed soonish.

_______________________ test_loop_started_in_constructor _______________________

cleanup = None

    def test_loop_started_in_constructor(cleanup):
        # test that SpecCluster.__init__ starts a loop in another thread
>       cluster = SpecCluster(worker_spec, scheduler=scheduler, loop=None)

../../BUILDROOT/python-distributed-2022.7.1-1.fc37.x86_64/usr/lib/python3.11/site-packages/distributed/deploy/tests/test_spec_cluster.py:87: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../BUILDROOT/python-distributed-2022.7.1-1.fc37.x86_64/usr/lib/python3.11/site-packages/distributed/deploy/spec.py:255: in __init__
    super().__init__(
../../BUILDROOT/python-distributed-2022.7.1-1.fc37.x86_64/usr/lib/python3.11/site-packages/distributed/deploy/cluster.py:67: in __init__
    self._loop_runner = LoopRunner(loop=loop, asynchronous=asynchronous)
../../BUILDROOT/python-distributed-2022.7.1-1.fc37.x86_64/usr/lib/python3.11/site-packages/distributed/utils.py:440: in __init__
    self._loop = IOLoop()
/usr/lib64/python3.11/site-packages/tornado/util.py:276: in __new__
    instance.initialize(*args, **init_kwargs)
/usr/lib64/python3.11/site-packages/tornado/platform/asyncio.py:339: in initialize
    super().initialize(**kwargs)
/usr/lib64/python3.11/site-packages/tornado/platform/asyncio.py:140: in initialize
    super().initialize(**kwargs)
/usr/lib64/python3.11/site-packages/tornado/ioloop.py:350: in initialize
    self.make_current()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <tornado.platform.asyncio.AsyncIOLoop object at 0x7fb8eff84690>

    def make_current(self) -> None:
>       warnings.warn(
            "make_current is deprecated; start the event loop first",
            DeprecationWarning,
            stacklevel=2,
        )
E       DeprecationWarning: make_current is deprecated; start the event loop first

/usr/lib64/python3.11/site-packages/tornado/platform/asyncio.py:353: DeprecationWarning

_____________________ test_client_async_before_loop_starts _____________________

cleanup = None

    def test_client_async_before_loop_starts(cleanup):
        async def close():
            async with client:
                pass
    
>       with pristine_loop() as loop:

../../BUILDROOT/python-distributed-2022.7.1-1.fc37.x86_64/usr/lib/python3.11/site-packages/distributed/tests/test_client.py:5540: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib64/python3.11/contextlib.py:137: in __enter__
    return next(self.gen)
../../BUILDROOT/python-distributed-2022.7.1-1.fc37.x86_64/usr/lib/python3.11/site-packages/distributed/utils_test.py:222: in pristine_loop
    IOLoop.clear_instance()
/usr/lib64/python3.11/site-packages/tornado/ioloop.py:227: in clear_instance
    IOLoop.clear_current()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    @staticmethod
    def clear_current() -> None:
        """Clears the `IOLoop` for the current thread.
    
        Intended primarily for use by test frameworks in between tests.
    
        .. versionchanged:: 5.0
           This method also clears the current `asyncio` event loop.
        .. deprecated:: 6.2
        """
>       warnings.warn(
            "clear_current is deprecated",
            DeprecationWarning,
            stacklevel=2,
        )
E       DeprecationWarning: clear_current is deprecated

/usr/lib64/python3.11/site-packages/tornado/ioloop.py:317: DeprecationWarning

What you expected to happen: Tests pass.

Environment:

• Dask version: 2022.7.1
• Python version: 3.11.0.b4
• Operating System: Fedora Rawhide
• Install method (conda, pip, source): source

[jrbourbeau]

Thanks @QuLogic! We're currently setting tornado <6.2 to avoid these deprecation errors / other possible unknown issues (xref https://github.com/dask/distributed/issues/6669)

https://github.com/dask/distributed/blob/a53858a5fc73b591da5a6ceff903a468990f8a21/requirements.txt#L12

I'm curious where this is showing up. Is while while packaging distributed?

[QuLogic]

Yes, I am trying to prepare packages for review in Fedora again, though there's lots of trouble in Rawhide at the moment due to new Tornado and Python 3.11 beta. I am a bit closer to passing on Fedora 36.

[hendrikmakait]

xref: #6669

[rikardn]

Using distributed==2022.12.0 and tornado==6.2 I still get the deprecation warning (and potentially some other issues).

_____________________________________________________________________________ test_allometry _____________________________________________________________________________
tests/integration/test_allometry.py:19: in test_allometry
    res = run_tool(
.tox/py310-integration/lib/python3.10/site-packages/pharmpy/tools/run.py:178: in run_tool
    return run_tool_with_name(name, tool, *args, **kwargs)
.tox/py310-integration/lib/python3.10/site-packages/pharmpy/tools/run.py:206: in run_tool_with_name
    res = execute_workflow(wf, dispatcher=dispatcher, database=database)
.tox/py310-integration/lib/python3.10/site-packages/pharmpy/workflows/execute.py:71: in execute_workflow
    res: T = dispatcher.run(workflow)
.tox/py310-integration/lib/python3.10/site-packages/pharmpy/workflows/dispatchers/local_dask.py:77: in run
    with LocalCluster(
.tox/py310-integration/lib/python3.10/site-packages/distributed/deploy/local.py:253: in __init__
    super().__init__(
.tox/py310-integration/lib/python3.10/site-packages/distributed/deploy/spec.py:259: in __init__
    super().__init__(
.tox/py310-integration/lib/python3.10/site-packages/distributed/deploy/cluster.py:70: in __init__
    self._loop_runner = LoopRunner(loop=loop, asynchronous=asynchronous)
.tox/py310-integration/lib/python3.10/site-packages/distributed/utils.py:449: in __init__
    self._loop = IOLoop()
.tox/py310-integration/lib/python3.10/site-packages/tornado/util.py:276: in __new__
    instance.initialize(*args, **init_kwargs)
.tox/py310-integration/lib/python3.10/site-packages/tornado/platform/asyncio.py:339: in initialize
    super().initialize(**kwargs)
.tox/py310-integration/lib/python3.10/site-packages/tornado/platform/asyncio.py:140: in initialize
    super().initialize(**kwargs)
.tox/py310-integration/lib/python3.10/site-packages/tornado/ioloop.py:350: in initialize
    self.make_current()
.tox/py310-integration/lib/python3.10/site-packages/tornado/platform/asyncio.py:353: in make_current
    warnings.warn(
E   DeprecationWarning: make_current is deprecated; start the event loop first
======================================================================== short test summary info =========================================================================
FAILED tests/integration/test_allometry.py::test_allometry - DeprecationWarning: make_current is deprecated; start the event loop first
=========================================================================== 1 failed in 1.35s ============================================================================
Exception ignored in atexit callback: <function close_clusters at 0x7ff1c3b120e0>
Traceback (most recent call last):
  File "/home/devel/pharmpy/.tox/py310-integration/lib/python3.10/site-packages/distributed/deploy/spec.py", line 677, in close_clusters
    cluster.close(timeout=10)
  File "/home/devel/pharmpy/.tox/py310-integration/lib/python3.10/site-packages/distributed/deploy/cluster.py", line 217, in close
    return self.sync(self._close, callback_timeout=timeout)
  File "/home/devel/pharmpy/.tox/py310-integration/lib/python3.10/site-packages/distributed/utils.py", line 332, in sync
    asynchronous = self.asynchronous
  File "/home/devel/pharmpy/.tox/py310-integration/lib/python3.10/site-packages/distributed/utils.py", line 325, in asynchronous
    return in_async_call(self.loop, default=getattr(self, "_asynchronous", False))
  File "/home/devel/pharmpy/.tox/py310-integration/lib/python3.10/site-packages/distributed/deploy/cluster.py", line 102, in loop
    self.__loop = loop = self._loop_runner.loop
AttributeError: 'LocalCluster' object has no attribute '_loop_runner'
Exception ignored in: <function Cluster.__del__ at 0x7ff1c3c1ecb0>
Traceback (most recent call last):
  File "/home/devel/pharmpy/.tox/py310-integration/lib/python3.10/site-packages/distributed/deploy/cluster.py", line 225, in __del__
ResourceWarning: unclosed cluster with a broken __repr__ <distributed.deploy.local.LocalCluster object at 0x7ff1c3b0ec80>
ERROR: InvocationError for command /home/devel/pharmpy/.tox/py310-integration/bin/pytest -vv tests/integration/test_allometry.py (exited with code 1)
________________________________________________________________________________ summary _________________________________________________________________________________
ERROR:   py310-integration: commands failed

[fjetter]

cc @graingert

[graingert]

@rikardn for your usecase could you grab the distributed.utils_test.loop_in_thread fixture and use that until we fix the LoopRunner?