tests/test_watch.py::test_awatch_interrupt_raise is failing with anyio>=4

[mgorny]

Description

When running the test suite against anyio-4.0.0 and 4.1.0, I'm getting the following failure:

$ python -m pytest
========================================================= test session starts =========================================================
platform linux -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0
rootdir: /tmp/watchfiles
configfile: pyproject.toml
testpaths: tests
plugins: timeout-2.2.0, mock-3.12.0, anyio-4.1.0
timeout: 10.0s
timeout method: signal
timeout func_only: False
collected 161 items                                                                                                                   

tests/test_cli.py .......................                                                                                       [ 14%]
tests/test_docs.py ................s                                                                                            [ 24%]
tests/test_filters.py ..................                                                                                        [ 36%]
tests/test_force_polling.py ..........................                                                                          [ 52%]
tests/test_run_process.py ..s.....................                                                                              [ 67%]
tests/test_rust_notify.py ....................................                                                                  [ 89%]
tests/test_watch.py ...............sF                                                                                           [100%]

============================================================== FAILURES ===============================================================
_____________________________________________________ test_awatch_interrupt_raise _____________________________________________________

self = <tests.test_watch.MockRustNotifyRaise object at 0x7f6e6c4fb090>
args = (1600, 50, 5000, <threading.Event at 0x7f6e6c4fafd0: set>)

    def watch(self, *args):
        if self.i == 1:
>           raise KeyboardInterrupt('test error')
E           KeyboardInterrupt: test error

tests/test_watch.py:204: KeyboardInterrupt

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/tmp/watchfiles/watchfiles/main.py", line 254, in awatch
    raw_changes = await anyio.to_thread.run_sync(watcher.watch, debounce, step, timeout, stop_event_)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/watchfiles/.venv/lib/python3.11/site-packages/anyio/to_thread.py", line 49, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/watchfiles/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2103, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/tmp/watchfiles/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 823, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/watchfiles/tests/test_watch.py", line 204, in watch
    raise KeyboardInterrupt('test error')
KeyboardInterrupt: test error

During handling of the above exception, another exception occurred:

  + Exception Group Traceback (most recent call last):
  |   File "/tmp/watchfiles/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 341, in from_call
  |     result: Optional[TResult] = func()
  |                                 ^^^^^^
  |   File "/tmp/watchfiles/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 262, in <lambda>
  |     lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  |             ^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/tmp/watchfiles/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
  |     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/tmp/watchfiles/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
  |     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/tmp/watchfiles/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 152, in _multicall
  |     return outcome.get_result()
  |            ^^^^^^^^^^^^^^^^^^^^
  |   File "/tmp/watchfiles/.venv/lib/python3.11/site-packages/pluggy/_result.py", line 114, in get_result
  |     raise exc.with_traceback(exc.__traceback__)
  |   File "/tmp/watchfiles/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
  |     res = hook_impl.function(*args)
  |           ^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/tmp/watchfiles/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 169, in pytest_runtest_call
  |     item.runtest()
  |   File "/tmp/watchfiles/.venv/lib/python3.11/site-packages/_pytest/python.py", line 1792, in runtest
  |     self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  |   File "/tmp/watchfiles/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
  |     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/tmp/watchfiles/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
  |     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/tmp/watchfiles/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 113, in _multicall
  |     raise exception.with_traceback(exception.__traceback__)
  |   File "/tmp/watchfiles/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
  |     res = hook_impl.function(*args)
  |           ^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/tmp/watchfiles/.venv/lib/python3.11/site-packages/anyio/pytest_plugin.py", line 124, in pytest_pyfunc_call
  |     runner.run_test(pyfuncitem.obj, testargs)
  |   File "/tmp/watchfiles/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 1928, in run_test
  |     self.get_loop().run_until_complete(
  |   File "/usr/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
  |     return future.result()
  |            ^^^^^^^^^^^^^^^
  |   File "/tmp/watchfiles/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 1888, in _call_in_runner_task
  |     return await future
  |            ^^^^^^^^^^^^
  |   File "/tmp/watchfiles/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 1866, in _run_tests_and_fixtures
  |     retval = await coro
  |              ^^^^^^^^^^
  |   File "/tmp/watchfiles/tests/test_watch.py", line 224, in test_awatch_interrupt_raise
  |     async for _ in awatch('.', stop_event=stop_event):
  |   File "/tmp/watchfiles/watchfiles/main.py", line 252, in awatch
  |     async with anyio.create_task_group() as tg:
  |   File "/tmp/watchfiles/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 658, in __aexit__
  |     raise BaseExceptionGroup(
  | BaseExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/tmp/watchfiles/watchfiles/main.py", line 254, in awatch
    |     raw_changes = await anyio.to_thread.run_sync(watcher.watch, debounce, step, timeout, stop_event_)
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/tmp/watchfiles/.venv/lib/python3.11/site-packages/anyio/to_thread.py", line 49, in run_sync
    |     return await get_async_backend().run_sync_in_worker_thread(
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/tmp/watchfiles/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2103, in run_sync_in_worker_thread
    |     return await future
    |            ^^^^^^^^^^^^
    |   File "/tmp/watchfiles/.venv/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 823, in run
    |     result = context.run(func, *args)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/tmp/watchfiles/tests/test_watch.py", line 204, in watch
    |     raise KeyboardInterrupt('test error')
    | KeyboardInterrupt: test error
    +------------------------------------
---------------------------------------------------------- Captured log call ----------------------------------------------------------
watchfiles.main INFO: 1 change detected
======================================================= short test summary info =======================================================
FAILED tests/test_watch.py::test_awatch_interrupt_raise - BaseExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
============================================== 1 failed, 157 passed, 3 skipped in 4.00s ===============================================

The relevant test passes with anyio-3.7.1.

Example Code

No response

Watchfiles Output

No response

Operating System & Architecture

Linux-6.6.3-gentoo-dist-x86_64-AMD_Ryzen_5_3600_6-Core_Processor-with-glibc2.38 #1 SMP PREEMPT_DYNAMIC Fri Dec 1 18:17:34 -00 2023

Environment

No response

Python & Watchfiles Version

python: 3.11.6 (main, Oct 26 2023, 14:30:39) [GCC 13.2.1 20231014], watchfiles: 0.0.0 (git 9f19a18da5323d0e63a2ee16c47ed6c819f89be5)

Rust & Cargo Version

cargo 1.73.0 rustc 1.73.0 (cc66ad468 2023-10-03) (gentoo)

[musicinmybrain]

I am seeing this while attempting to upgrade anyio to 4.2.0 in Fedora Linux (Rawhide). The packaged watchfiles version is currently 0.20.0.

[musicinmybrain]

I understand that this library still pins anyio to <4:

https://github.com/samuelcolvin/watchfiles/blob/9f19a18da5323d0e63a2ee16c47ed6c819f89be5/setup.py#L24

Is there a plan for supporting anyio 4.x?

[musicinmybrain]

I understand that this library still pins anyio to <4:

https://github.com/samuelcolvin/watchfiles/blob/9f19a18da5323d0e63a2ee16c47ed6c819f89be5/setup.py#L24

Is there a plan for supporting anyio 4.x?

Oops, no, it doesn’t—not in pyproject.toml, at least!

https://github.com/samuelcolvin/watchfiles/blob/9f19a18da5323d0e63a2ee16c47ed6c819f89be5/pyproject.toml#L13

[samuelcolvin]

I've created a pr to delete setup.py to avoid confusion.

[bnavigator]

Please also pin anyio in pyproject.toml or even better make it compatible with the anyio 4 changes:

https://anyio.readthedocs.io/en/stable/versionhistory.html

BACKWARDS INCOMPATIBLE Changes to cancellation semantics: Any exceptions raising out of a task groups are now nested inside an ExceptionGroup (or BaseExceptionGroup if one or more BaseException were included)

[verga]

Same problem to update opensuse tumbleweed: Problem: the to be installed python311-watchfiles-0.21.0-2.1.x86_64 requires '(python311-anyio >= 3.0.0 with python311-anyio < 4)', but this requirement cannot be provided

[bnavigator]

Same problem to update opensuse tumbleweed: Problem: the to be installed python311-watchfiles-0.21.0-2.1.x86_64 requires '(python311-anyio >= 3.0.0 with python311-anyio < 4)', but this requirement cannot be provided

That's why I createed python-anyio3: https://build.opensuse.org/request/show/1145404 Wait for the next snapshot.

[musicinmybrain]

This is one of the last incompatibilities giving us trouble while trying to upgrade the python-anyio and python-trio packages to recent versions in Fedora Rawhide, and (via python-uvicorn and python-sphinx-autobuild) the most serious in terms of the number of impacted packages.

Is there any prospect of a patch to support anyio 4 in the relatively near future?