pytest-django
pytest-django copied to clipboard
pytest-dev
Using the same database with `xdist` and `--create-db` results in error.
We are trying to avoid creating databases for each xdist process since our migrations are taking a very long time (They have network calls. I know, I know). The pytest-django documentation at https://pytest-django.readthedocs.io/en/latest/database.html?highlight=xdist#use-the-same-database-for-all-xdist-processes suggests it would be possible to use xdist and let processes rely on the same database.
But in reality, there is a race condition when the same database is tried to be used, and it causes a bunch of "duplicate * value" errors.
Is there a recommended way around this problem? Thanks!
Here is a sample error:
pytest.ini
[pytest]
addopts =
--cov=hyke
--cov-report=html
--cov-report=term
--cov-report=xml
--failed-first
--no-cov-on-fail
--numprocesses=auto
--reuse-db
bash
pipenv run pytest --create-db
============================= test session starts ==============================
platform linux -- Python 3.9.10, pytest-7.1.1, pluggy-1.0.0
django: settings: **omitted** (from option)
rootdir: **omitted**, configfile: pytest.ini
plugins: cov-3.0.0, Faker-13.3.4, forked-1.4.0, xdist-2.5.0, django-4.5.2, ddtrace-0.60.2
gw0 I / gw1 I
gw0 [1614] / gw1 [1614]
_______ ERROR at setup of **omitted** ________
[gw1] linux -- Python 3.9.10 **omitted**/bin/python
self = <django.db.backends.utils.CursorWrapper object at 0x7f4a84c39550>
sql = 'CREATE DATABASE "test_postgres" ', params = None
ignored_wrapper_args = (False, {'connection': <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7f4a84c3[90](https://github.com/**omitted**/runs/6013133846?check_suite_focus=true#step:5:90)70>, 'cursor': <django.db.backends.utils.CursorWrapper object at 0x7f4a84c3[95](https://github.com/**omitted**/runs/6013133846?check_suite_focus=true#step:5:95)50>})
def _execute(self, sql, params, *ignored_wrapper_args):
self.db.validate_no_broken_transaction()
with self.db.wrap_database_errors:
if params is None:
# params default might be backend specific.
> return self.cursor.execute(sql)
E psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "pg_database_datname_index"
E DETAIL: Key (datname)=(test_postgres) already exists.
I think pytest-django needs a logic that would make sure the database is created for the first connection, and the other processes just shared a connection to that DB, similar to what is being done at Django's DiscoverRunner: https://github.com/django/django/blob/main/django/test/utils.py#L196
And that logic needs to be implemented at https://github.com/pytest-dev/pytest-django/blob/master/pytest_django/fixtures.py#L101
To be clear, I am referring to the code section within if parallel > 1:. Just a theory though.
Actually no. I guess you are already doing it manually in your code at django_db_modify_db_settings_xdist_suffix. Hmm 😕
I guess the default Django-test runner's parallel option does not even allow re-using the same DB on different processes.
I am guessing we need some kind of a mutex so that one of the runners calls setup_databases while others wait for it. I am not sure if that is possible through what pydest-xdist provides though.
Through trial and error, I have realized an underlying problem with our migrations having network calls. It somehow was causing the test run to start running before applied migrations. I think something was erroring out and breaking a wait loop.