pytest-split
pytest-split copied to clipboard
jerry-git
Using xdist with pytest-split
Since most CI machines has between one and a half to two CPUs it is a waste not to combine both approaches when appropriate. Can we use both of these plugins together?
Good question! I tried it with my demo repo which uses GitHub Actions, I think they have 2 CPUs available. Seems to work nicely https://github.com/jerry-git/pytest-split-gh-actions-demo/pull/1.
Here's the CI run: https://github.com/jerry-git/pytest-split-gh-actions-demo/runs/549132036. Group 1 contains 6 tests which each take around 10 seconds. With -n 2 it runs in around 30 seconds.
That's strange it takes so long. It might be a bug or a performance issue in this plugin.
Anyway, xdist is more efficient when there are significantly more tests.
FWIW, I ran using pytest-xdist and --splits 3 on https://github.com/python/mypy as I thought it was a good sample of many tests, most fast, some slow. Using pytest-split I got a total time of 357.85s.
Meanwhile running bare pytest takes 240-250s total. We also have a script where we split up the tests manually into faster/slower groups, and the run time was 153.54s.
@thedrow The demo project contains dummy tests which basically just sleep for a fixed number of seconds. Thus the results I explained are aligned with my expectations.
@ethanhs That sounds very surprising. pytest-split should not really add any notable overhead. Could you describe a bit more what kind of setup you used? Did you have the test durations file available (pytest-split seems to currently run all tests if that's not available)? If I understood correctly, you used both pytest-xdist and pytest-split? Did you run locally or with some CI system? If I looked correctly, the test step seems to currently take around 18-19 minutes in mypy GitHub Actions workflow (Windows executor).
@jerry-git I think part of the issue is that test collection/setup has significant overheard, especially on manycore machines. I was running this on my desktop, which is a Ryzen 3900X 12 core CPU, on Windows.
@ethanhs I just cloned the mypy repo to see what's going on. So, I assume by "bare pytest" you mean pytest + pytest-xdist with -nauto option (uses all available cores to parallelise the work). -nauto is configured here https://github.com/python/mypy/blob/43461c199ada319fc0e83c021886b040aa458a73/pytest.ini#L23. On my 16-core machine it takes around 290 seconds with that configuration. I assume your trial with pytest-split had also -nauto in place and pytest-xdist installed? I.e. three python processes:
pytest --splits 3 --group 1 -nauto
pytest --splits 3 --group 2 -nauto
pytest --splits 3 --group 3 -nauto
Combining pytest-split and pytest-xdist in a single machine does not really bring any value as -nauto already makes sure that all available resources are used as efficiently as possible. So it does not make sense to split the work further with pytest-split.
Mypy would likely get a significant boost at least in GitHub Actions (I assume would be doable with Travis as well) based jobs if you split the work with pytest-split to multiple parallel jobs and use pytest-xdist within each job (like you already do). I can see from the logs that currently those GitHub Actions based windows tests use 2 cores, I assume that's all what's available as mypy has that -nauto configured. For example, if you split the work with pytest-split to e.g. 4 parallel jobs, I assume the build time will be close to 4x faster. Example configuration for parallel jobs: https://github.com/jerry-git/pytest-split-gh-actions-demo/blob/9fffe751c0a718e455a32a6601d82a38c379cc9e/.github/workflows/test.yml#L8-L22
I might even fork it and try it myself 🤔
Oh thank you for looking into that! Yeah I was using pytest split with xdist -nauto.
We currently have a script to run the tests in groups because the xdist scheduler is naive, and it means that we end up getting long running tests at the end of runs. I was hoping by grouping we could avoid that issue, but it looks like that may not work out.
Using multiple actions workers definitely seems like a promising choice, I'll look into that. Thanks!
@ethanhs I ran some experiments locally. I think it's not behaving well locally if -n auto is used for each group. If I distribute the work to CPUs based on the amount of groups, the results are better. I have 16 CPUs available.
267.40s: pytest -n 16
258.61s: pytest --splits 2 --group 1 -n 8 & pytest --splits 2 --group 2 -n 8
232.10s: pytest --splits 4 --group 1 -n 4 & pytest --splits 4 --group 2 -n 4 & pytest --splits 4 --group 3 -n 4 & pytest --splits 4 --group 4 -n 4
So, for me it looks like pytest-split is providing some value also in single machine case when pytest-xdist is also in use. And most importantly, pytest-split does not seem to add any notable overhead to e.g. test collection based on these results.
@jerry-git You are correct that in CI environments -nauto should spawn two workers.
I think the reason for this behavior you're describing is that each group has even execution times (hopefully).
@jerry-git, interesting, thank you for sharing that data! Are you running on Linux, macOS, or Windows?
Yeah it seems that running things split in parallel does indeed speed things up! That is heartening to see. Great work :)
I have macOS. I think it'd be interesting to do some trials with mypy test suite in a single machine environment in which there are no other heavy processes consuming the resources of the CPUs. This was not the case in my local experiment. I'd assume that the gain of using pytest-split + pytest-xdist vs only pytest-xdist could be more significant in such case.
However, such environment is not very interesting consider most real life use cases as we usually have either the local environment (single machine, multiple CPUs, some heavy processes running on some CPUs) or CI environment (multiple, close to identical "machines" which each has 1 or 2 CPUs available, no other heavy processes running). The main value pytest-split brings is the possibility to split the workload evenly to multiple "machines".
Thanks @jerry-git for creating this project. Would running pytest-xdist with --store-durations work together?
