loadscope on parametrized classes not going parallel

[manubansal]

I am parametrizing a test class and scheduling tests with --dist loadscope. I was expecting the effect that every parameter combination will be treated as a separate class by pytest. Then, those parametrized runs should be able to go to different pytest processes when using loadscope. However, I am not seeing that behavior. All runs stay on the same process always. There doesn't seem to be any different in scheduling behavior when using loadscope between parametrizing a test method versus parametrizing a test class.

What is the expected behavior with loadscope and parametrized classes? And if they are supposed to run in parallel, is this a bug?

(lux-D-BxQKsC-py3.8) ➜  test cat testp.py
import pytest

@pytest.mark.parametrize("paramvalue", ["p1", "p2"])
class TestClass:
  def test_method_1(self, paramvalue):
    assert True

  def test_method_2(self, paramvalue):
    assert True
(lux-D-BxQKsC-py3.8) ➜  test pytest -n auto --dist loadscope -vvv testp.py
==================================================== test session starts ====================================================
platform darwin -- Python 3.8.12, pytest-6.2.4, py-1.11.0, pluggy-0.13.1 -- /Users/manub/Library/Caches/pypoetry/virtualenvs/lux-D-BxQKsC-py3.8/bin/python
cachedir: .pytest_cache
rootdir: /Users/manub/tmp/test
plugins: xdist-2.5.0, forked-1.4.0, rerunfailures-10.2, mock-3.7.0, dotenv-0.5.2, profiling-1.7.0
[gw0] darwin Python 3.8.12 cwd: /Users/manub/tmp/test
[gw1] darwin Python 3.8.12 cwd: /Users/manub/tmp/test
[gw2] darwin Python 3.8.12 cwd: /Users/manub/tmp/test
[gw3] darwin Python 3.8.12 cwd: /Users/manub/tmp/test
[gw4] darwin Python 3.8.12 cwd: /Users/manub/tmp/test
[gw5] darwin Python 3.8.12 cwd: /Users/manub/tmp/test
[gw0] Python 3.8.12 (default, Nov 29 2021, 23:59:10)  -- [Clang 11.0.3 (clang-1103.0.32.62)]
[gw1] Python 3.8.12 (default, Nov 29 2021, 23:59:10)  -- [Clang 11.0.3 (clang-1103.0.32.62)]
[gw2] Python 3.8.12 (default, Nov 29 2021, 23:59:10)  -- [Clang 11.0.3 (clang-1103.0.32.62)]
[gw3] Python 3.8.12 (default, Nov 29 2021, 23:59:10)  -- [Clang 11.0.3 (clang-1103.0.32.62)]
[gw4] Python 3.8.12 (default, Nov 29 2021, 23:59:10)  -- [Clang 11.0.3 (clang-1103.0.32.62)]
[gw5] Python 3.8.12 (default, Nov 29 2021, 23:59:10)  -- [Clang 11.0.3 (clang-1103.0.32.62)]
gw0 [4] / gw1 [4] / gw2 [4] / gw3 [4] / gw4 [4] / gw5 [4]
scheduling tests via LoadScopeScheduling

testp.py::TestClass::test_method_1[p1] 
[gw0] [ 25%] PASSED testp.py::TestClass::test_method_1[p1] 
testp.py::TestClass::test_method_1[p2] 
[gw0] [ 50%] PASSED testp.py::TestClass::test_method_1[p2] 
testp.py::TestClass::test_method_2[p1] 
[gw0] [ 75%] PASSED testp.py::TestClass::test_method_2[p1] 
testp.py::TestClass::test_method_2[p2] 
[gw0] [100%] PASSED testp.py::TestClass::test_method_2[p2] 

===================================================== 4 passed in 0.86s =====================================================
(lux-D-BxQKsC-py3.8) ➜  test 

[rcomer]

If I have understood, loadscope relies on the string generated by pytest to determine how to group the tests.

https://github.com/pytest-dev/pytest-xdist/blob/61132777f8d85f7e03837684fbbdb41cb4f9c486/src/xdist/scheduler/loadscope.py#L270-L292

So to get this to work, I think we would need a change in pytest itself so that

testp.py::TestClass::test_method_1[p1] 

becomes

testp.py::TestClass[p1]::test_method_1

[xunzhou]

temporary workaround https://github.com/pytest-dev/pytest-xdist/compare/master...xunzhou:pytest-xdist:master

[rcomer]

@xunzhou that seems promising for the parametrized class case discussed above, but would it also modify cases where an individual method on a class is parametrized?

[xunzhou]

You meant something like this?

class TestClass1:
    def test_case1(self):
        pass

    @pytest.mark.parametrize('param', [1])
    def test_case2(self, param):
        pass

still want both test_case1 and test_case2 schedule to same worker since they are under same class right?

[rcomer]

I was thinking something like

class TestClass1:
    @pytest.mark.parametrize('param', ["foo", "bar"])
    def test_case(self, param):
        pass

and I would expect both the "foo" and "bar" test to go to the same worker.

But I also agree that with your example I would expect test_case1 and test_case2 to go to the same worker.