warnings.catch_warnings is async-unsafe

[fstirlitz]

import warnings
import asyncio

async def spam():
	with warnings.catch_warnings(record=True) as ws:
		await asyncio.sleep(0.1)
		w = Warning("12345")
		warnings.warn(w)
		assert w in (ww.message for ww in ws)

async def ham():
	with warnings.catch_warnings(record=True) as ws:
		await asyncio.sleep(0.2)

async def main():
	await asyncio.gather(spam(), ham())

asyncio.run(main())

Despite technically being documented (https://docs.python.org/3/library/warnings.html#warnings.catch_warnings), this is very error-prone, surprising and not at all useful. The warnings module should probably use contextvars to manage the warning handler.

[graingert]

See also https://github.com/python/cpython/issues/81785 and https://github.com/python/cpython/issues/50896

[kumaraditya303]

Using contextvars makes sense here.

[gvanrossum]

Sounds like a big project.

[joaoe]

Hi. I've looked into the warnings module code. I so happens that in my project warnings, logging and pytest don't cooperate too well.

The warning module stores pointers to the original warning functions and swaps that function in and out with the catch_warnings() context mnager. This is obviously a recipe for disaster either running with threads or just if mixing catch_warnings() with other APIs that also like to take over the warnings function.

My suggestion is that all monkeypatching of global functions in the warnings module must be removed. I have a draft implementation I can push eventually, which uses contextvars to keep track of which catch_warnings() object is active and push the warning messages there. And that's it is. So far my branch is incomplete because it also needs to keep track of filters activated with catch_warnings().

Hopefully we can have a discussion about the implementation and potentially consider including the new implentation in the next major release.

[gvanrossum]

@joaoe Please describe your design and explain how it solves the problem. I'm very interested in hearing from you!

[joaoe]

Hi @gvanrossum surely :)

Here's a draft commit https://github.com/joaoe/cpython/commit/thread_safe_warnings Notes:

• there is a global stack of catch_warnings() and active filters, protected by a RLock. Usable by libraries that want to take over all warnings, like test frameworks and logging.captureWarnings().
• there is also a similar thread local stack using contextvars, to fix the problem presented in the original post.
• the stack keeps track of which catch_warnings() objects have been created and in which order. From the stack traversing downwards, one can find the current filters and who consumes messages
• the commit is a compromise between keeping some old behavior and adding new one, e.g., still having a global filters list.
• with this implentation, global symbols in the warnings module, like showwarning and filters become "read-only" meaning, it shouldn't actualy be supported to swap these objects in and out to control the behavior of the module. This is implemented with the Protect object in the end, but a final implementation would most likely have some different implementation, e.g., printing a deprecation warning?
• the commit also fixes logging.captureWarnings(). If called with a catch_warnings() in context (this happens with pytest), it would break.
• the commit touches no C code although that would need to be fixed as well
• I fixed a number of tests. Many other tests fail for me, but if I extract some tests to the side and run the code separately, it works. Some shenanigans with the test framework perhaps or my environment setup. Skipping "import _warnings" fixed some tests and broke others.

Again, this is a draft, a suggestion. Hopefully this helps bringing this problem forward.

[gvanrossum]

Hi @joaoe, I'm afraid I won't personally have time to review this any time soon. I hope that another core dev or triager might be interested. Maybe @kumaraditya303, @serhiy-storchaka, or @sobolevn ? Also, I recommend that you open a Draft PR -- that's easier for reviewers. The title of the draft PR should start with "gh-91505:" so it will be linked back to this issue. Good luck!