database error with pytest-django and pytest-xdist 1.26.1

[SpoonMeiser]

trafficstars

pytest with pytest-django is giving us errors tearing down databases since upgrading to pytest-xdist 1.26.1. This happens nearly 100% of the time, but very occasionally doesn't. I suspect that having a large number of tests exacerbates the problem.

We get an error like this (anonymised):

_________________________________________________________________________________________________________________________________________________________________ ERROR at teardown of TestClass.test_some_behaviour _________________________________________________________________________________________________________________________________________________________________
[gw12] linux2 -- Python 2.7.15 /home/user/virtualenv/venv/bin/python2

    def teardown_database():
        with django_db_blocker.unblock():
>           teardown_databases(db_cfg, verbosity=request.config.option.verbose)

../../virtualenv/venv/local/lib/python2.7/site-packages/pytest_django/fixtures.py:113: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../virtualenv/venv/local/lib/python2.7/site-packages/pytest_django/compat.py:14: in teardown_databases
    db_cfg
../../virtualenv/venv/local/lib/python2.7/site-packages/django/test/runner.py:509: in teardown_databases
    connection.creation.destroy_test_db(old_name, self.verbosity, self.keepdb)
../../virtualenv/venv/local/lib/python2.7/site-packages/django/db/backends/base/creation.py:264: in destroy_test_db
    self._destroy_test_db(test_database_name, verbosity)
../../virtualenv/venv/local/lib/python2.7/site-packages/django/db/backends/base/creation.py:283: in _destroy_test_db
    % self.connection.ops.quote_name(test_database_name))
../../virtualenv/venv/local/lib/python2.7/site-packages/django/db/backends/utils.py:64: in execute
    return self.cursor.execute(sql, params)
../../virtualenv/venv/local/lib/python2.7/site-packages/django/db/utils.py:95: in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <django.db.backends.utils.CursorWrapper object at 0x7f0c769703d0>, sql = 'DROP DATABASE "test_pytest_gw12"', params = None

    def execute(self, sql, params=None):
        self.db.validate_no_broken_transaction()
        with self.db.wrap_database_errors:
            if params is None:
>               return self.cursor.execute(sql)
E               OperationalError: database "test_pytest_gw12" is being accessed by other users
E               DETAIL:  There are 10 other sessions using the database.

../../virtualenv/venv/local/lib/python2.7/site-packages/django/db/backends/utils.py:62: OperationalError

I have been running with pytest-random and the --random argument to test this behaviour, and the name of the test that failed is different every time, and I don't think relevant to this behaviour; I don't think this is a problem with any of the tests. This problem occurs without the pytest-random plugin installed - I was just using that to convince myself that the problem wasn't related to a particular test suite.

Sometimes an additional failure with as IntegrityError will occur on the primary key of a record when inserting into the database, but I think this a symptom of the above error rather than a legitimate test error.

The commandline used is:

pytest --ds=test_settings_local --disable-warnings -n auto --random

This problem started occurring around the time pytest-xdist 1.26.1 was released, and downgrading to pytest-xdist 1.26.0 makes the problem (seem to) go away, which leads me to believe that the problem is with pytest-xdist 1.26.1

[SpoonMeiser]

With further investigation, I now think this problem also exists when using version 1.26.0. That was the only thing I could find that had changed that might be related to this problem, so now I'm no longer sure if this problem is directly related to xdist or not.

For the time being, we've stopped using xdist altogether, and the problem doesn't occur. I'm happy for this to be closed and I can raise again if I investigate further in the future and narrow down the problem better, unless anyone thinks they could make any progress with the information I have provided so far.

[RonnyPfannschmidt]

there seems to be more than 10 still existing users of the database, i suspect that xdist is not related, but makes the issue so much worse that it happens every time

[SpoonMeiser]

pytest xdists should make a separate database (IIUC) for each worker, so there should only be one user of each database. I don't understand what situation could occur that would increase that number.

It seems as though the number of users of the database is loosely related to the number of workers (it differs on each run)

[RonnyPfannschmidt]

@SpoonMeiser its possibly related to the pytest upgrade where we moved xunit setup/teardown into the fixture system, i recall pytest-django replicating part o that, thus being triggered into incorrect behavior, i suggest testing against a lower pytest version just to verify that idea

[SpoonMeiser]

Thanks, it's super useful to just get another suggestion of something to test, I'll test that and get back to you.

[blueyed]

@SpoonMeiser its possibly related to the pytest upgrade where we moved xunit setup/teardown into the fixture system, i recall pytest-django replicating part o that, thus being triggered into incorrect behavior, i suggest testing against a lower pytest version just to verify that idea

Yeah, pytest-django skips its handling of unittests with pytest 4.2+, but there appear to be issues with that (i.e. it is not really compatible / the same).

Are you using unittest / Django's TestCase etc?

[ruohola]

Any update on this?

[shacker]

Pretty sure I'm looking at the same bug here, so adding my notes. We have a very db-intensive test suite (needs to be for this app). We use Postgres, pytest, pytest-django, and pytest-xdist on MacOS. Previously, we could run our test suite with -n 8 and the parallelism would work just fine; 8 databases were created and destroyed. But lately we've been seeing a lot of errors like the one described here. Specifically:

                self.log('Got an error creating the test database: %s' % e)
>               sys.exit(2)
E               SystemExit: 2

I've done some testing and come up with the example below, in case it's helpful debugging this.

Decorating any test with @pytest.mark.django_db invokes db creation, even if no data is created or touched in the test body. Note that this sample test does not use any code from our project -- it's "bare":

# sometest.py

import pytest

@pytest.mark.django_db
def test_a():
    assert 1 == 1


@pytest.mark.django_db
def test_b():
    assert 1 == 1

• If I run pytest sometest.py it works just fine.
• If I run pytest -n 2 sometest.py it fails with a database creation error similar to the one above.
• If I comment out one of the two decorators and again run pytest -n 2 sometest.py it's fine again.

In other words, the problem only comes up when xdist is invoked AND the pytest collector finds more than one test that touches the database.

Current testrunner libs are:

plugins: testinfra-1.16.0, sugar-0.9.4, html-3.1.1, cov-2.11.1, metadata-1.11.0, xdist-2.2.1, faker-2.0.0, Faker-4.1.2, django-4.1.0, flakefinder-1.0.0, forked-1.3.0

Going on other suggestions, I've tried downgrading python-xdist to 1.34.0 but it did not fix the problem.

It is not clear to me whether the problem is in pytest-django or in python-xdist. I'm going to cross-post this to issues on both projects.

pytest-django issue is here.

[cat-turner]

I seem to have this issue as well

[danmash]

In my case, it was caused by the wrong config for databases:

DATABASES = {
    'default': {
        'NAME': 'postgres',
    },
    'caching_database': {
        'NAME': 'postgres',
    },
    'user_db': {
        'NAME': 'postgres',
    }
}

So pytest-xdist tried to create a new database using the database name test_postgres_gw0 three times and as a result, failed with SystemExit: 2