django-nose `manage.py --parallel` flag has no effect under django-nose

Django 1.9 introduced the --parallel flag, to run tests in parallel. This gives a 4x speedup on my 4-core Macbook Pro under the default unittest runner, which is pretty great.

Unfortunately I get no effect when I use this flag with the nose test runner.

It does look like the setup is being performed:

$ ./manage.py test --parallel 4
nosetests --logging-clear-handlers --verbosity=1
Creating test database for alias 'default'...
Cloning test database for alias 'default'...
Destroying old test database for alias 'default'...
mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
Cloning test database for alias 'default'...
mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
Cloning test database for alias 'default'...
mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
Cloning test database for alias 'default'...
mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
.......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 743 tests in 68.034s

OK
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...

But the test duration is no better than without the --parallel flag.

Is this flag expected to work? I don't see any documentation for it, though it is explicitly passed through to manage.py here https://github.com/django-nose/django-nose/blob/3b9dad77d0440cace471aa43d77a4ba619f145bb/django_nose/runner.py#L91

Nov 04 '16 05:11 paultiplady

Good question. I have not used this option myself, so I'm not sure how it is supposed to work.

There may be additional options needed to tell nose that you want to run in multi-process mode. You may need to pass processes=4 or the equivalent setting:

http://nose.readthedocs.io/en/latest/usage.html?highlight=processes#cmdoption--processes

You may also need to look carefully at your test setup, to see if you can mark fixtures and test classes as being available for multiprocess testing:

http://nose.readthedocs.io/en/latest/doc_tests/test_multiprocess/multiprocess.html

Nov 07 '16 13:11 jwhitlock

I had previously tried the processes=4 flag, and it crashes and burns violently:

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
ERROR: test suite for <class 'user.tests.test_views.TestUserViews'>
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/django/db/backends/mysql/base.py", line 112, in execute
    return self.cursor.execute(query, args)
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/MySQLdb/cursors.py", line 250, in execute
    self.errorhandler(self, exc, value)
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/MySQLdb/connections.py", line 42, in defaulterrorhandler
    raise errorvalue
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/MySQLdb/cursors.py", line 247, in execute
    res = self._query(query)
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/MySQLdb/cursors.py", line 411, in _query
    rowcount = self._do_query(q)
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/MySQLdb/cursors.py", line 374, in _do_query
    db.query(q)
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/MySQLdb/connections.py", line 270, in query
    _mysql.connection.query(self, query)
_mysql_exceptions.OperationalError: (2006, 'MySQL server has gone away')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/nose/plugins/multiprocess.py", line 788, in run
    self.setUp()
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/nose/suite.py", line 293, in setUp
    self.setupContext(ancestor)
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/nose/plugins/multiprocess.py", line 770, in setupContext
    super(NoSharedFixtureContextSuite, self).setupContext(context)
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/nose/suite.py", line 316, in setupContext
    try_run(context, names)
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/nose/util.py", line 471, in try_run
    return func()
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/django/test/testcases.py", line 1021, in setUpClass
    if not connections_support_transactions():
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/django/test/testcases.py", line 986, in connections_support_transactions
    for conn in connections.all())
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/django/test/testcases.py", line 986, in <genexpr>
    for conn in connections.all())
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/django/utils/functional.py", line 33, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/django/db/backends/base/features.py", line 226, in supports_transactions
    cursor.execute('CREATE TABLE ROLLBACK_TEST (X INT)')
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/django/db/utils.py", line 95, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/django/db/backends/mysql/base.py", line 112, in execute
    return self.cursor.execute(query, args)
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/MySQLdb/cursors.py", line 250, in execute
    self.errorhandler(self, exc, value)
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/MySQLdb/connections.py", line 42, in defaulterrorhandler
    raise errorvalue
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/MySQLdb/cursors.py", line 247, in execute
    res = self._query(query)
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/MySQLdb/cursors.py", line 411, in _query
    rowcount = self._do_query(q)
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/MySQLdb/cursors.py", line 374, in _do_query
    db.query(q)
  File "/Users/paul/.virtualenvs/backend/lib/python3.4/site-packages/MySQLdb/connections.py", line 270, in query
    _mysql.connection.query(self, query)
django.db.utils.OperationalError: (2006, 'MySQL server has gone away')

----------------------------------------------------------------------
Ran 0 tests in 0.151s

This looks suspiciously like a Rails mysql parallelization issue here: http://stackoverflow.com/questions/5269876/activerecordstatementinvalid-mysqlerror-mysql-server-has-gone-away-using

I.e. the Mysql driver might not be naively parallelizable, and we need to wait until after forking to establish DB connections. (I could believe this is the magic implemented in the django --parallel flag).

Setting _multiprocess_can_split_ = True without --processes doesn't have any effect on the test time, though it does actually run successfully.

Nov 07 '16 19:11 paultiplady

Was this with both? Like ./manage.py test --parallel 4 --processes 4?

Nov 07 '16 19:11 jwhitlock

Same result both with --processes=4 alone, and combined with --parallel=4.

Nov 07 '16 20:11 paultiplady

This appears to be affection Django 1.9 but not Django 1.10:

$ python manage.py test --parallel
nosetests --verbosity=1
Creating test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
.......................^C^C
----------------------------------------------------------------------
Ran 24 tests in 8.472s

OK
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...

Nov 07 '16 22:11 makyo

I get the same results on django 1.10 as on 1.9:

$ ./manage.py test --parallel=4
nosetests --logging-clear-handlers --verbosity=1
Creating test database for alias 'default'...
Cloning test database for alias 'default'...
mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
Cloning test database for alias 'default'...
mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
Cloning test database for alias 'default'...
mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
Cloning test database for alias 'default'...
mysqldump: [Warning] Using a password on the command line interface can be insecure.
mysql: [Warning] Using a password on the command line interface can be insecure.
.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 729 tests in 65.055s

OK
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
$ ./manage.py test
nosetests --logging-clear-handlers --verbosity=1
Creating test database for alias 'default'...
.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 729 tests in 64.861s

OK
Destroying test database for alias 'default'...

Note the test durations are the same.

Nov 08 '16 00:11 paultiplady

@paultiplady I stand corrected, thank you :)

$ venv/bin/python manage.py test administration
nosetests administration --verbosity=1
Creating test database for alias 'default'...
...........................................................................................
----------------------------------------------------------------------
Ran 91 tests in 72.177s

OK
Destroying test database for alias 'default'...

$ venv/bin/python manage.py test --parallel=4 administration
nosetests administration --verbosity=1
Creating test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
...........................................................................................
----------------------------------------------------------------------
Ran 91 tests in 70.456s

OK
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...

$ venv/bin/python manage.py test --parallel --testrunner django.test.runner.DiscoverRunner administration
Creating test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
...........................................................................................
----------------------------------------------------------------------
Ran 91 tests in 41.972s

OK
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...
Destroying test database for alias 'default'...

Nov 08 '16 01:11 makyo

+1 I'm hitting the same issue as well. Any one have any luck with fixing this?

Jan 31 '17 18:01 bqumsiyeh

After adding up configuration, fixing some tests to make it completely work in nose, I found running test is taking 2x time than django's default runner, and that's when I found this issue :(

I think this issue must be mentioned in the docs.

Anyway, I'm not here just to whine. I spent some time to figure out the reason. So, this issue won't have any workaround other than proper fix in the django-nose code. We can't use processes because of the reason mentioned by @paultiplady and also all the processes would be sharing the same DB which would cause migrations/fixtures issue.

I don't have time to fix it up, I'll probably be switching back to django's default runner. Here's some implementation detail in case anyone wants to fix it up:

The way django's default runner handles this problem is, it partitions the whole testsuite into sub-suites[0] and abstract those multiple suites by subclassing unittest.TestSuite, naming it as ParallelTestSuite[1] . After that, it passes that suite to unittest.TextTestRunner which calls the run method of the suite. In that run[2] method, it creates pool of processes which when initialised, changes the db connections[3] and runs the tests. So, all the processes work upon different db.

To fix this issue in django-nose, the key idea is one will have to do similar stuff of creating multiple processes here[4], run all the sub-suites, keep accumulating results and return it.

[0] https://github.com/django/django/blob/d3449faaa915a08c275b35de01e66a7ef6bdb2dc/django/test/runner.py#L676

[1] https://github.com/django/django/blob/d3449faaa915a08c275b35de01e66a7ef6bdb2dc/django/test/runner.py#L313

[2] https://github.com/django/django/blob/d3449faaa915a08c275b35de01e66a7ef6bdb2dc/django/test/runner.py#L340

[3] https://github.com/django/django/blob/d3449faaa915a08c275b35de01e66a7ef6bdb2dc/django/test/runner.py#L275

[4] https://github.com/django-nose/django-nose/blob/347a711934688feb1d04cd4b17f8aafba995b241/django_nose/runner.py#L244

Aug 12 '18 15:08 Ashish-Bansal

FWIW I'm using pytest to run my tests these days, and pytest-xdist supports parallelization nicely. I got a 4x speed boost on my UTs, and have completely removed nose from my project.

I found that nose.parameterized did not play well even with django's native parallelization, whereas pytest.parametrize does work.

The main complaint I have about pytest is that you still need to use Django's TestCase if you want to do setUpTestData style caching of models between tests (which gives a great speedup); pytest fixtures don't play well with django models when you set them at session scope. It would be more idiomatic to just use test functions with named fixtures for sharing your test data, but that doesn't work right now: https://github.com/pytest-dev/pytest-django/issues/514.

Aug 13 '18 18:08 paultiplady

FWIW it is working for me now. I am using --keepdb so I avoid the migration problem. If I need to do a migration then I run it without --parallel or --keepdb

Just --parallel (no argument) which defaults to number of processors.

nose==1.3.7
django-nose==1.4.5

Oct 24 '18 15:10 crucialfelix

django-nose django-nose copied to clipboard

`manage.py --parallel` flag has no effect under django-nose

django-nose
django-nose copied to clipboard