pytest-xdist

pytest-xdist copied to clipboard

Currently when tests are run on multiple cores the name of the files are not displayed any more. It would be rather useful to be able to still receive the same information even if it means one need to wait until all the end of the test run.

Without using numprocesses:

collected 114 items / 28 skipped / 86 selected                                                                                                                    

astropy/io/misc/yaml.py .                                                                                                                                   [  0%]
astropy/io/misc/tests/test_hdf5.py ........................................................ssss...........ss.                                               [ 65%]
astropy/io/misc/tests/test_pandas.py .....                                                                                                                  [ 70%]
astropy/io/misc/tests/test_pickle_helpers.py ....                                                                                                           [ 73%]
astropy/io/misc/tests/test_yaml.py ..............................                                                                                           [100%]

With numprocesses:

rootdir: /Users/bsipocz/munka/devel/astropy, inifile: setup.cfg
plugins: xdist-1.29.0, forked-1.0.2, arraydiff-0.3, cov-2.6.0, xonsh-0.8.3, dependency-0.4.0, doctestplus-0.3.0, requests-mock-1.6.0, openfiles-0.4.0.dev0, remotedata-0.3.2.dev0
gw0 C / gw1 C/usr/local/lib/python3.7/site-packages/pytest_doctestplus/plugin.py:10: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
/usr/local/lib/python3.7/site-packages/pytest_doctestplus/plugin.py:10: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
gw0 [114] / gw1 [114]1 skipped                                                                                                                                    
...................................................ssss.............s.s...........................................                                          [100%]

Also, having the info of the total number of collected and skipped tests is useful, too, and it's not printed in the latter case either.

Hi @bsipocz,

If you pass -v you get a report line for each test. It is more verbose but does what you want, I believe.

the current terminal ux is not sanely able to show more than one file in progress at the same time

Yes, -v is way too verbose. I see that doing this real time is problematic, but having a summary report that looks similar to the normal run would be great. I realize that making this happen is most likely not that simple either.

Yeah the xdist reporter would need to be able to jump back and forth, the regular report is missing useful information and the -v is hard to read (as it shows test ids both to say that it's started and to provide the status). That would probably have to be an additional plugin though, the runner can't really assume it has a proper term with all the flexibility of a TUI I think?

I'm a bit confused:

• when running pytest, I get pretty output with the test file name and the progress percentage.
• when running pytest -n auto I get . and E output -- no test file names or progress percentage
• but when I add pytest-pretty, I get the test file name back

It seems like pytest-xdist is overriding the report output, but since it seems to work with pytest-pretty I'm not sure why. Is pytest-pretty giving incorrect output, or is pytest-xdist changing the report output when that isn't necessary?

Ah, I think I understand -- it looks like pytest-pretty is outputting:

foo/test_foo.py . [1%]
foo/test_bar.py . [1%]
foo/test_foo.py . [2%]
foo/test_bar.py .. [2%]

I can imagine why pytest-xdist changed the report output, but can the pytest-xdist report changes be disabled? I'd like to use the default pytest report output, but I'm unsure how.

I can imagine why pytest-xdist changed the report output, but can the pytest-xdist report changes be disabled?

xdist would have to jump back and forth vertically e.g. update the progress of test_foo after it's already written the line for test_bar. For the normal mode this is not an issue because it runs every file linearly so test_bar is only written out once test_foo has been completed, so everything is just written left to right and top to bottom.

Per @RonnyPfannschmidt's comment the internal reporting API does not currently have the ability to perform such effects (which also requires the ability to detect the capabilities or lack thereof of the terminal), and would likely need significant updates before it's a possibility.

Were xdist's changes to the reporting system disabled it would just throw unreadable garbage at the screen e.g.

astropy/io/misc/yaml.py astropy/io/misc/tests/test_hdf5.py astropy/io/misc/tests/test_pandas.py ............................................................ssss...........ss......

or something even worse.

I would assume a similar issue occurs when using one of the randomising plugins and randomising across files, and their solution is similar.

It's probably reasonably easy to do this with rich tables these days, however nobody has stepped up to do it

There are multiple rich based plugins for ux that can experiment with this

Were xdist's changes to the reporting system disabled it would just throw unreadable garbage at the screen e.g.

Do you have any suggestions on how I might test this?

Nope, execnet and xdist interleave in ways where I have to read Int the code myself to figure how to turn things off

In purely practical terms it means N+1 pytest processes writing to stdout with no coordination