progressive don't play well with nose.plugin.multiprocess

[aht]

progressive can only run test sequentially, it seems.

--processes=8--with-progressive 50 tests [...] in 53.6s

--processes=8 [...] 50 tests in 13.544s

I am using nose 1.1.2 and nose-progressive 0.7, tests are run with django-nose.

[erikrose]

Thanks for the report! I'll have a look.

[erikrose]

Wow, your tests are pretty CPU-intense. We're going to start doing multiprocess at Mozilla in the next few months, so I'm also interested in having this work.

[aht]

Actually my tests are I/O intense, but network I/O, so the more processes waiting, the better.

[erikrose]

Ah, gotcha.

[dlanger]

Using nose-progressive==1.3 (nose=1.1.2 and django-nose=1.1) I'm seeing similar:

./manage.py [...] --processes=4 --with-progressive     57.57s user 5.22s system 99% cpu 1:03.11 total       
./manage.py [...] --processes=4    120.13s user 8.80s system 326% cpu 39.524 total

[dlanger]

Reproducible at https://github.com/dlanger/nose-test-case

[erikrose]

Factoring Django out…

(py27)[12:34:50 ~/Checkouts/more-itertools]% time nosetests --processes=2 ~/Desktop/timetests.py
..
----------------------------------------------------------------------
Ran 2 tests in 10.036s
OK
nosetests --processes=2 ~/Desktop/timetests.py  0.20s user 0.08s system 2% cpu 10.239 total

(py27)[12:35:59 ~/Checkouts/more-itertools]% time nosetests --with-progressive --processes=2 ~/Desktop/timetests.py
OK!  2 tests, 0 failures, 0 errors in 15.0s
nosetests --with-progressive --processes=2 ~/Desktop/timetests.py  0.12s user 0.04s system 1% cpu 15.165 total

…where timetests.py is…

import time
from unittest import TestCase


class SimpleTest(TestCase):
  _multiprocess_shared_ = True 

  def test_sleep_10(self):
    time.sleep(10)

  def test_sleep_5(self):
    time.sleep(5)

[erikrose]

Hmm, this makes it look like the nose framework is hostile to incremental reporting of multiprocess results:

"As each test or suite executes in a worker process, results (failures, errors, and specially handled exceptions like SkipTest) are collected in that process. When the worker process finishes, it returns results to the main nose process. There, any progress output is printed (dots!), and the results from the test run are combined into a consolidated result set." (https://nose.readthedocs.org/en/latest/plugins/multiprocess.html)

[erikrose]

Of course, we should be able to at least not stymie parallelization.

[mathrick]

Any progress on this? At the very least, a mention in the docs would be nice to make it clear that progressive does not currently play nicely with parallelisation. Of course, getting it to play nicely would be even better...