Warning from tracer causes Flaky

[jobh]

If removing pytest.warns here on py3.12,

https://github.com/HypothesisWorks/hypothesis/blob/e51d473907120b3c9b441279b83b9cf549c4df80/hypothesis-python/tests/cover/test_monitoring.py#L35

I'd expect a warning to be reported. Instead, an uninformative Flaky exception follows:

[...]
    self.conclude_test(Status.OVERRUN)
  File "/home/jobh/src/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/data.py", line 2631, in conclude_test
    raise StopTest(self.testcounter)
hypothesis.errors.StopTest: 2

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  [...]
File "/home/jobh/src/hypothesis/hypothesis-python/src/hypothesis/core.py", line 1300, in _raise_to_user
    raise the_error_hypothesis_found
hypothesis.errors.Flaky: Unreliable assumption: An example which satisfied assumptions on the first run now fails it.
============================================================================================== slowest 20 durations ===============================================================================================

(3 durations < 1s hidden.  Use -vv to show these durations.)
============================================================================================= short test summary info =============================================================================================
FAILED tests/cover/test_monitoring.py::test_monitoring_warns_on_registered_tool_id - hypothesis.errors.Flaky: Unreliable assumption: An example which satisfied assumptions on the first run now fails it.
================================================================================================ 1 failed in 0.10s ================================================================================================

(https://github.com/HypothesisWorks/hypothesis/actions/runs/9692069470/job/26744718765?pr=4023)

I guess this can be fixed by f.x. re-raising all warnings as fatal here, https://github.com/HypothesisWorks/hypothesis/blob/e51d473907120b3c9b441279b83b9cf549c4df80/hypothesis-python/src/hypothesis/core.py#L1064

But beyond that: The consequence of using pytest.warns in internal tests is that the exceptions are never raised, so we don't see the consequences of mis-handling them. Maybe we should always use pytest.raises instead, to ensure the full machinery is excercised.

[jobh]

The consequence of using pytest.warns in internal tests is that the exceptions are never raised, so we don't see the consequences of mis-handling them. Maybe we should always use pytest.raises instead, to ensure the full machinery is excercised.

Ideally, we should check both the warn-and-continue and the raise-warning behaviours. The first part would verify that the warning isn't about anything that leads to another error down the road. See WIP PR #4023 for example