Incompatible scopes of live_server and transactional_db

[aaugustin]

The live_server fixture is session-scoped while the transactional_db fixture (which live_server implicitly enables) is function-scoped.

As a consequence, live_server can handle requests during the whole duration of the test suite. These requests can collide with the database flushes that transactional_db performs after each test or with the periods when access to the database is disabled between two tests.

Here's what the sequence of events looks like:

• Session: live_server fixture starts a LiveServerThread
  • Function: transactional_db fixture runs (triggered by the function-scoped, autouse _live_server_helper fixture) and adds TransactionTestCase._post_teardown to finalizers
  • Function: test makes a request to the live server, perhaps from a selenium-driven browser, and terminates before the live server can respond to that request (this can happen for various reasons; a common one is accessing a page that makes AJAX requests when it loads towards the end of the test and exiting the test function before all these requests have completed)
  • Function: transactional_db finalizer runs and attempts to flush the database, conflicting with the requests the live server is still processing
• Session: live_server finalizer waits for LiveServerThread to terminate

The conflict can cause deadlocks between any SQL query from a HTTP request handled by the live server and the query that flushes the database, which looks like this:

Exception Database test_xxxxxxxx couldn't be flushed. Possible reasons:
  * The database isn't running or isn't configured correctly.
  * At least one of the expected database tables doesn't exist.
  * The SQL was invalid.
Hint: Look at the output of 'django-admin sqlflush'. That's the SQL this command wasn't able to run.
The full error: deadlock detected
DETAIL:  Process 37253 waits for AccessExclusiveLock on relation 131803 of database 131722; blocked by process 37250.
Process 37250 waits for AccessShareLock on relation 132659 of database 131722; blocked by process 37253.
HINT:  See server log for query details.

This failure mode is particularly annoying because the database isn't flushed, requiring to next test run to re-create the database and thus negating the benefits of --reuse-db.

If database reuse isn't enabled, destroying the test database can fail with:

django.db.utils.OperationalError: database "test_xxxxxxxx" is being accessed by other users
DETAIL:  There is 1 other session using the database.

I've also seen SQL queries from HTTP requests handled by the live server to fail with:

Failed: Database access not allowed, use the "django_db" mark, or the "db" or "transactional_db" fixtures to enable it.

because the transactional_db finalizer has flushed the database and blocked access until the next test.

I'm planning to look for workarounds and will update this ticket with my findings.

[b4handjr]

Running into this as well. Did you find anything?

[aaugustin]

Nope.

I'll report my findings here if I find time to work on a solution. It isn't near the top of my TODO list.

[mpasternak]

I'm having the same(?) problem, live_server fixture does not see changes made to the database, running in it's own transaction. @aaugustin , does it sound likely to you?

I was wondering... maybe I should ditch the live_server fixture totally and just write my replacement. It would consist of running "manage.py runserver" in a thread. Except pointing it to a test database, I'd not update project settings. At some point I may need that. Also, LiveServerTestCase does not support custom STATICFILES_FINDERS except the filesystem one...

[Helumpago]

I'm also running into this issue. Current workaround I'm using is to override the live_server fixture so that it is function scoped. Specifically, my conftest.py file contains the following:

from pytest_django.fixtures import live_server as orig_live_server

@pytest.fixture(scope='function')
def live_server(request):
    """
        Workaround inspired by https://github.com/mozilla/addons-server/pull/4875/files#diff-0223c02758be2ac7967ea22c6fa4b361R96
    """
    return orig_live_server(request)

This seems like it's working, though admittedly, it slows all the test cases that use it way down. If anyone sees a problem with this approach, please let me know.

[andreymal]

I have similar problem: live_server is not working, it prints error no such table: django_session.

The @Helumpago's workaround fixed this error for me. But pytest prints this warning:

RemovedInPytest4Warning: Fixture "live_server" called directly. Fixtures are not meant to be called directly, are created automatically when test functions request them as parameters.

Is there any other way to use live_server with a database?

[andreymal]

The @Helumpago's workaround raises an error instead of warning since pytest 4.0 (2018-11-13) :(

UPD: looks like it works (but this is ugly)

from pytest_django.fixtures import live_server as orig_live_server
live_server = pytest.fixture(scope='function')(orig_live_server.__wrapped__)

[mmcardle]

I have had some success with the code below, but (YMMV)

@pytest.fixture(name='live_server', scope='function')
def _live_server(live_server):
    # This rescopes the live_server fixture to function scope
    # See https://github.com/pytest-dev/pytest-django/issues/454
    return live_server

scope='function' is not technically required as the default scope is function but is added for clarity

[silviogutierrez]

I'm running into this too. No doubt all the above variants of the fix will work.

But I'm wondering: what is the correct behavior here? It seems like, the root problem is actually those AJAX requests still happening after the test finishes. Rightly so, they should prevent the database from flushing.

Think about it in real life and not a test. If you had a DB and wanted to run a flush command, and had a ton of active requests towards that table, wouldn't you expect an error or to have to handle it, rather than it just working and magically killing all those pending requests?

I can think of a few options:

1. Have your Selenium client wait for all outstanding AJAX requests to finish before ceding control. I wrap my Selenium usage so this is fairly straightforward, and my app has a way to signal loading is done. But this is impractical to many. Especially so for apps that "poll" in the background. There may always be an outstanding request.
2. Somehow finagle pytest/flush commands to kill all outstanding processes to the database. Something like this psql postgres -c 'SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE pid <> EVERYTHING_BUT_PYTEST_PID';
3. Change scope, but as pointed out, this slows down everything.
4. Maybe there's a magical way Selenium do #1 in a more structured, universal way. Something like: wait for all pending AJAX requests to finish but start no new ones. I'm not familiar enough with Selenium to verify this right now.

[atcrank]

I'm not sure if I'm running into this problem or if it is just a well known issue that the docs are hinting at, but after I request a 'live_server' fixture for a test, I do not get the normal db fixture (which I've overloaded to ingest a file of test data) on subsequent tests. I tried mmcardle's suggested wrapper for re-scoping live_server down to 'function', but the problem remained. Putting all live_server tests last is sort of doable with alphabetically ordering the test filenames, but don't feel great about tests that work in one order and not another.

[NathanSmeltzer]

If the above fixes do not work for anyone else, try temporarily disabling the --reuse-db option (usually in your pytest.ini file) and running the test(s). If it no longer errors, add the --reuse-db option back.

[shulcsm]

Are there any new developments or better workaround? Waiting for requests to settle works but it's slow and not very reliable. Terminating or canceling query backend does not work since that just bubbles up the exception in server thread.