Custom test grouping and test group order logic

[SalmonMode]

Closes #18

It's been long requested (#18) to allow for custom logic for the distribution of tests among test groups in a simple and clean way. This adds that functionality by providing a hook that each tests nodeid is passed to, and whatever is returned by the hook for a given test is that test's test group name. All tests with the same test group name will be batched together and passed to the same worker together.

I also added another hook that grants access to the finished collection of test groups so that the order of the groups can be changed as needed. This allows for certain optimizations to be made, such as putting slow tests in the same group and placing that group in the beginning of the collection in order to speed the test suite up.

I also took the opportunity to clean up the logic quite a bit as there was a lot of repetition between the different load schedulers. Now they are unified into the same base class (except for EachScheduling) with the only differing logic being what their default grouping logic is. I felt I should leave EachScheduling alone for now, but I imagine it could also use the same base class by simply having every test in the same group, and then duplicating that group N times. But that seemed kinda memory heavy.

Thanks for submitting a PR, your contribution is really appreciated!

Here's a quick checklist that should be present in PRs:

• [x ] Make sure to include reasonable tests for your change if necessary

• [x ] We use towncrier for changelog management, so please add a news file into the changelog folder following these guidelines:

  • Name it $issue_id.$type for example 588.bugfix;

  • If you don't have an issue_id change it to the PR id after creating it

  • Ensure type is one of removal, feature, bugfix, vendor, doc or trivial

  • Make sure to use full sentences with correct case and punctuation, for example:

    Fix issue with non-ascii contents in doctest text files.
    

[nicoddemus]

Hi @SalmonMode,

Thanks a lot for the PR! As you mention this is a long-requested feature so all the work you have put into it is more than welcome!

I like the approach of providing "low level" hooks that lets users and plugins customize test grouping to be sent to workers. Also so far in the review I'm very satisfied with the code quality and documentation you've put into it. 👍

I'm writing this just so to let you know that it will probably take awhile to review a PR of this magnitude, so please be patient! 😁

[SalmonMode]

@nicoddemus haha I appreciate it :grin: . Take your time. I spent a very long time on it (granted, at least half of that time was on what to name the hooks).

As a side note, the name I chose was because I figure at some point in the future, this may be expanded to allow grouping based on markers, rather than just names, but I couldn't figure out how to pull those in in a good way, and wanted to get the PR in rather than holding onto it for longer.

[ssbarnea]

Any chance to refresh this one?

[nicoddemus]

Any chance to refresh this one?

Thanks for the ping @ssbarnea! Indeed I had lost sight of this.

Will make time for completing my review this week. 👍

[skhomuti]

@nicoddemus just reminding about the existence of this PR :)

[Wilsontomass]

Heya @nicoddemus, sorry for the bump but we would be extremely grateful for a PR of this kind to be added to pytest-xdist. Is it still feasable? Has some other functionality superseded it?

[nicoddemus]

Thanks a lot for the ping!

I had left a bunch of comments, but forgot to publish them 🤦 :

I just published them, I will give it another review later.

[nicoddemus]

we should add a hook that allows to inform the coordinator about affinities (session scope fixtures, certain configurations)

Just so we are in the same page, you mean in this PR or in a follow up?