Error on teardown of test database in Django integration test

[medihack]

trafficstars

When running an integration (sync) test using Pytest in Django I end up with a warning that another session is using the database:

adit/core/tests/test_tasks.py::test_process_dicom_task_that_succeeds
  /app:0: PytestWarning: Error when trying to teardown test databases: OperationalError('database "test_postgres" is being accessed by other users\nDETAIL:  There is 1 other session using the database.')

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

This happens as soon as I run the worker, as mentioned in the documentation:

my_app = app.with_connector(app.connector.get_worker_connector())  # type: ignore
my_app.run_worker(wait=False, install_signal_handlers=False, listen_notify=False)

It seems the worker leaves some connections open. I tried calling app.close() explicitly at the end of the test, but this also doesn't help.

[medihack]

It's more complicated than I thought. I also run some async stuff in my sync task, and this error doesn't seem to purely depend on Procrastinate. I will close this for now and investigate it further. Also, our mix of sync and async stuff really bites us quite often 🙄.

[medihack]

I investigated this a bit further. The above error happens as soon as I do a database access in the task, e.g.:

@app.task
def example_task():
    job = ProcrastinateJob.objects.first()
    assert job is not None

Also when using async tasks, e.g.:

@app.task
async def example_task_async():
    job = await ProcrastinateJob.objects.afirst()
    assert job is not None

A simple workaround is to close the database connection at the end of a task manually by calling db.close_old_connections() resp. await database_sync_to_async(db.close_old_connections)(). It doesn't work to close the connections in the test itself; it must be in the task. And maybe there is also a nicer workaround (without changing the task's code).

Maybe we should add a hint about this to the documentation. That's why I reopen this issue.

[ewjoachim]

This looks like we might be able to make a reproducing test in procrastinate's own codebase ?

[medihack]

Sure, I can add an integration test. It won't fail; pytest just provides it as a warning, but it will be visible on the console.

[ewjoachim]

You can turn that warning into an error with @pytest.mark.filterwarnings (you can use filterwarning for the whole testsuite but in this cas we probably want to do it only for the specific tests that handle this)

[medihack]

Just to keep you informed, it is not reproducible in Procrastinate itself, but we use a slightly different setup (Django 5, Psycopg 3). I will set up a minimal Django 5 project with Procrastinate in the upcoming days and see if I can reproduce it there, but I have the feeling there's nothing Procrastinate can do about it. I will keep you updated, and eventually close the issue again.

[mschfh]

Was this ever resolved?

Running into this as well (Django 5.1, psycopg 3) when starting a worker as documented, both with run_worker in the task and also with the fixture.