cupy icon indicating copy to clipboard operation
cupy copied to clipboard

BUG,TST: Fix `cupy_tests/core_tests` runs for `pytest-run-parallel`

Open seberg opened this issue 2 months ago • 3 comments

Thought a small PR is maybe good, this is start to test with pytest-run-parallel. The cupy_tests/core_tests (mostly?) run with this. I did free-threading earlier, but for now it doesn't really matter, these are just threading problems in the tests.

Some notes:

  • Must use python 3.14, because the mock things (mostly?) work threadsafe there (also warning filters).
  • Avoid global rng state (_helper) fix.
  • SetUp/TearDown are not thread-safe, but for some cases setup_class/teardown_class seems OK and good enough for pytest-run-parallel.
    • In general, have to avoid state on the test class now.
  • Some tests just can't work, start marking them with @pytest.mark.thread_unsafe

For future reference, I ran this ASAN (parts with TSAN at an earlier point) with:

CUPY_TEST_GPU_LIMIT=8 ASAN_OPTIONS=detect_leaks=0,protect_shadow_gap=0 pytest "cupy_tests/core_tests" -n8 --parallel-threads=20 -s -vv -k "not TestRawJitify and not TestRawJitifyJitify"

(Jitify tests are not just slow, I was using ASAN and that ran into issues with any failed test. -- in practice mostly run without any test_raw.py.)

seberg avatar Nov 17 '25 16:11 seberg

This has grown a bit. I think I may have occasional hangs (I am hoping this is due to the known memory allocator deadlock), and nothing is 100% (testing is slow, but naturally we are testing for race conditions to begin with).

But, this now makes cupy_tests/core_tests pass with free-threading. I do have further fixes beyond this, but I think this should go in first (some of this may be complicated enough).

seberg avatar Dec 10 '25 14:12 seberg

@seberg let's fix the pre-test failure and kick off CI!

leofang avatar Dec 10 '25 22:12 leofang

OK, the fact that instantiating has to be deferred makes this bit more tedious (I actually needed a critical_section in free-threaded). But let's give it a shot now...

/test mini

seberg avatar Dec 11 '25 14:12 seberg