4.11.0 test database is not created and used but main database is "cleared"

[sshishov]

Hi we were using the same setup for years but recent upgrade to 4.11.1 broke everything. The problem that instead of creating test_<db> it is loading "session" fixtures inside main db and flushes it as we are doing call_command

Rolling back to 4.10.0 fixes the issue and test_<db> is created successfully.

This is part of our setup:

def run_sql(query: str, fetch: bool = False, db: str = 'default') -> tuple[t.Any, ...] | None:
    conn = psycopg.connect(
        user=ADMIN_USER, password=ADMIN_PASSWORD, host=django_settings.DATABASES[db]['HOST'], port=django_settings.DATABASES['default']['PORT']
    )
    (cur := conn.cursor()).execute(query)
    response: tuple[t.Any, ...] | None = cur.fetchone() if fetch else None
    conn.close()
    return response


@pytest.fixture(scope='session')
def django_db_createdb(request: pytest.FixtureRequest, django_db_createdb: bool) -> bool:
    db_name = f'test_{django_settings.DATABASES["default"]["NAME"]}'
    if xdist_suffix := getattr(request.config, 'workerinput', {}).get('workerid'):
        db_name = f'{db_name}_{xdist_suffix}'
    db_exists = (result := run_sql(query=f"SELECT EXISTS (SELECT 1 FROM pg_database WHERE datname='{db_name}')", fetch=True)) and result and result[0]
    if django_db_createdb or not db_exists:
        run_sql('CREATE EXTENSION IF NOT EXISTS vector')
    return django_db_createdb or not db_exists


@pytest.fixture(scope='session')
def django_db_setup(django_db_setup: None, django_db_blocker: pytest_django.DjangoDbBlocker, django_db_createdb: bool) -> None:
    del django_db_setup
    if django_db_createdb:
        with django_db_blocker.unblock():
            call_command('flush', '--noinput')
            call_command('loaddata', *pathlib.Path().glob('tests/db_fixtures/**/*.yaml'))

run_sql is just custom raw SQL runner to execute random queries. And then we have django_db_createdb and django_db_setup where we override something, decide something and also flushing test db and loading fixtures.

Somehow now everything is executed on our main <db> instead of test_<db>

I guess you can easily reproduce it using these 2 fixtures without non needed stuff

[kingbuzzman]

..More details from another issue

[kingbuzzman]

@sshishov can you give me any more details? I replicated your environment here https://github.com/pytest-dev/pytest-django/pull/1240 and the test works as expected. Could there be any other functions/fixtures you're using? Please check and let me know. Thanks!

[kingbuzzman]

@sshishov any updates? Can i close this?

[sshishov]

Hi @kingbuzzman , I will create today the fresh repo and try to reproduce there. If it is not reproduced, then it seems we have some "incorrectly" used fixtures or something... Will update you shortly. Just gimme some time, do not close the ticket yet.

[maingoh]

I am facing the same issue since a recent dependency bump, reverted to 4.10.0 in the meantime. We use this fixture

@pytest.fixture(scope="session", autouse=True)
def enable_db_access_for_all_tests(django_db_setup, django_db_blocker):
    """
    Unblock the db for all tests
    """
    with django_db_blocker.unblock():
        yield

[maingoh]

The issue seems to be in django_db_blocker because if the test is marked with pytest.mark.django_db the test db is created/used.

[sshishov]

Hi guys, checking it now... it is working for me... Maybe some specific case caused this issue. If it will reoccur, will let you know. Maybe some additional package or additional env-variable set was causing it... Cannot tell for sure now.

Current setup:

• django: 5.2.8
• pytest-django: 4.11.1

[kingbuzzman]

@maingoh is this still an issue for you? Otherwise I'll close it.

[maingoh]

Well, still an issue on my side (which is normal since no new release or fix has been done). I still pin the old version and use django 4.2 (maybe it makes a difference, but I cannot easily bump to see if it fixes the issue). I unfortunately don't have much time to debug, but it seems that something has been broken.

@sshishov did you change the dependencies (django ?) in the meantime ?

[kingbuzzman]

So just to confirm, your tests do not have @pytest.mark.django_db and instead you rely on this

@ pytest.fixture(scope="session", autouse=True) def enable_db_access_for_all_tests(django_db_setup, django_db_blocker): """ Unblock the db for all tests """ with django_db_blocker.unblock(): yield

that i'm guessing is at your root conftest.py, to give access to the db for all your tests? Did I get anything wrong? Is there anything i didn't mention/left out? Is there anything else you're doing that was not mentioned?

[maingoh]

that i'm guessing is at your root conftest.py, to give access to the db for all your tests? Did I get anything wrong? Is there anything i didn't mention/left out? Is there anything else you're doing that was not mentioned?

Exactly, it's hard to tell if I am doing anything more as it's a quite big codebase. I will try to find some time to reproduce with a more minimalist code structure but cannot garantee when :(

So just to confirm, your tests do not have @pytest.mark.django_db and instead you rely on this

I do have some tests with @pytest.mark.django_db and when executing only those it is working. If executing the full directory, I guess it depends on which test is first executed (if one with @pytest.mark.django_db it will create the db, otherwise it will not)

[maingoh]

Looks like it comes from this commit https://github.com/pytest-dev/pytest-django/commit/8000db04f07822861331d0df8ef52f9c67eafc00 The aliases returned are empty {}. I think in my case it should be None to allow all databases.

It goes there https://github.com/pytest-dev/pytest-django/blob/cbfaba0d7bcfa7b8cae49d73d9aa486b854ff96a/pytest_django/fixtures.py#L137

In the changelog it says

To fix this, decorate at least one test with the :func:`django_db <pytest.mark.django_db>` marker with appropriate ``databases`` and ``serialized_rollback`` settings.`

But it does not work when selecting specific tests.

The default of setup_databases is too allow all databases by default, pytest-django should have the same behavior https://docs.djangoproject.com/fr/4.2/topics/testing/advanced/#django.test.utils.setup_databases