Possibility to split tests on Class basis

[adaptivecodes]

I am currently using this plugin for splitting up my tests. A lot of my pytests consist of multiple test cases which are depended on one another. Is there a possibility to split the tests on class basis? Example: Testclass A: test_01: test_02: Testclass B: test_01: test_02: test_03: Testclass C: test_01: Splitting those in two would do something like this: Batch A: A(test_01, test_02) C(test_01) Batch B: B (test_01, test_02, test_03)

Currently I am getting something like this: Batch A: A(test_01, test_02) B(test_01) Batch B: B(test_02, test_03), C(test_01)

Maybe I am missing something. If this is not possible at the moment are you willing to accept a Pull Request?

Greetings, Andreas

[christianbundy]

Adding my use-case here: Django's TestCase runs one transaction per class, so it's optimal to run each class on a single worker.

[jerry-git]

Happy to take a PR for this 🙂 I this the best would be to have a CLI flag for it such that it could be used with all the current (and future) splitting algorithms. Another option would be to implement a new splitting algorithm which is tailored for this class use case.

[long-git-hash]

I am currently working on an implementation for that.

[jerry-git]

FYI there's also demand for keeping tests from a single module within the same group: https://github.com/jerry-git/pytest-split/issues/108. Thus, I think introducing a CLI flag for both use cases is better than new splitting algorithms. For https://github.com/jerry-git/pytest-split/issues/108 I suggested --no-module-splits and for this one I think --no-class-splits would make sense. I believe there are plenty of synergies for these two features considering what kind of changes are needed in the current implementation.

[long-git-hash]

My current implementation goes this way, that I want to support the following splitting boundaries:

class Boundary(enum.Enum):
    FUNCTION = pytest.Function
    CLASS = pytest.Class
    MODULE = pytest.Module
    PACKAGE = pytest.Package
    FILE = pytest.File
    DIR = pytest.Dir

And I thought about a parameter called boundary supporting the values

• function
• class
• module
• package
• file
• dir

[long-git-hash]

While implementing the PR for this "splitting on boundaries" feature I stumbled over an issue with the test coverage.

I get the warning:

~/git/pytest-split/.venv/lib/python3.12/site-packages/coverage/inorout.py:519: CoverageWarning: Module pytest_split was previously imported, but not measured (module-not-measured)
  self.warn(msg, slug="module-not-measured")

And in the coverage report below I see that the first lines of each file are always not covered, which (from my limited understanding) should usually not be the case, because they are only imports and definitions.

---------- coverage: platform linux, python 3.12.3-final-0 -----------
Name                                      Stmts   Miss  Cover   Missing
-----------------------------------------------------------------------
src/pytest_split/__init__.py                  0      0   100%
src/pytest_split/algorithms.py               83     30    64%   1-33, 36, 42-59, 114-127, 155, 167, 176, 185-190
src/pytest_split/cli.py                      13      0   100%
src/pytest_split/folding.py                  47     41    13%   1-26, 30, 36-46, 56-93
src/pytest_split/ipynb_compatibility.py      27      5    81%   1-9, 35, 47, 57
src/pytest_split/plugin.py                   95     23    76%   1-25, 87-88, 116, 129-130, 152-153, 165-166, 185, 196, 215-220
-----------------------------------------------------------------------
TOTAL                                       265     99    63%

I don't have an idea how to resolve this. From what I read online it might be connected to the fact, that this plugin somehow uses an old version of the pytest plugin testing API. Any suggestions how to remove this warning (and increase our test coverage)?

[long-git-hash]

@jerry-git What shall I do with this test coverage issue mentioned in https://github.com/jerry-git/pytest-split/issues/82#issuecomment-2569566564 (just above)? Shall I submit my changes with some minor refactoring in tests or in separate commits?

[mkotzjan]

@jerry-git any update on this topic?

[long-git-hash]

My custom version splitting on class boundaries is now running for half a year in our test rack.