Connection already closed

[sweetpythoncode]

After the first test runner, the connection to DB dropped. django.db.utils.InterfaceError: connection already closed Seems to work with pytest-django==4.2.0

But broken with:

pytest==6.2.4
pytest-django==4.4.0
psycopg2-binary==2.9.1

Error log:

self = <django.db.backends.postgresql.base.DatabaseWrapper object at 0x7f66ebdddb80>
name = None

    @async_unsafe
    def create_cursor(self, name=None):
        if name:
            # In autocommit mode, the cursor will be used outside of a
            # transaction, hence use a holdable cursor.
            cursor = self.connection.cursor(name, scrollable=False, withhold=self.connection.autocommit)
        else:
>           cursor = self.connection.cursor()
E           django.db.utils.InterfaceError: connection already closed

../../../../../env/lib/python3.8/site-packages/django/db/backends/postgresql/base.py:236: InterfaceError


args = (<django.db.backends.postgresql.base.DatabaseWrapper object at 0x7f66ebdddb80>, None)
kwargs = {}
event_loop = <_UnixSelectorEventLoop running=False closed=False debug=False>

    @functools.wraps(func)
    def inner(*args, **kwargs):
        if not os.environ.get('DJANGO_ALLOW_ASYNC_UNSAFE'):
            # Detect a running event loop in this thread.
            try:
                event_loop = asyncio.get_event_loop()
            except RuntimeError:
                pass
            else:
                if event_loop.is_running():
                    raise SynchronousOnlyOperation(message)
        # Pass onwards.
>       return func(*args, **kwargs)

[bluetech]

Does your project use multiple databases, or creative fixtures involving the database?

[xykylikuf001]

Same problem:

    self = <django.contrib.gis.db.backends.postgis.base.DatabaseWrapper object at 0x7fde9030a820>, name = None
    
        def _cursor(self, name=None):
            self.ensure_connection()
            with self.wrap_database_errors:
    >           return self._prepare_cursor(self.create_cursor(name))
    
    venv/lib/python3.9/site-packages/django/db/backends/base/base.py:237: 
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    
    args = (<django.contrib.gis.db.backends.postgis.base.DatabaseWrapper object at 0x7fde9030a820>, None), kwargs = {}
    event_loop = <_UnixSelectorEventLoop running=False closed=False debug=False>
    
        @functools.wraps(func)
        def inner(*args, **kwargs):
            if not os.environ.get('DJANGO_ALLOW_ASYNC_UNSAFE'):
                # Detect a running event loop in this thread.
                try:
                    event_loop = asyncio.get_event_loop()
                except RuntimeError:
                    pass
                else:
                    if event_loop.is_running():
                        raise SynchronousOnlyOperation(message)
            # Pass onwards.
    >       return func(*args, **kwargs)
    
    venv/lib/python3.9/site-packages/django/utils/asyncio.py:26: 
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    
    self = <django.contrib.gis.db.backends.postgis.base.DatabaseWrapper object at 0x7fde9030a820>, name = None
    
        @async_unsafe
        def create_cursor(self, name=None):
            if name:
                # In autocommit mode, the cursor will be used outside of a
                # transaction, hence use a holdable cursor.
                cursor = self.connection.cursor(name, scrollable=False, withhold=self.connection.autocommit)
            else:
    >           cursor = self.connection.cursor()
    E           psycopg2.InterfaceError: connection already closed
    
    venv/lib/python3.9/site-packages/django/db/backends/postgresql/base.py:236: InterfaceError
    
    The above exception was the direct cause of the following exception:
    
    self = <backend.contrib.order.tests.test_tasks.TestOrderTask object at 0x7fde878a3b50>, celery_worker = <Worker: gen138507@vip-virtual-machine (running)>
    
        def test_set_profile_orders_total_count(self, celery_worker):
    >       profile = UserModel.objects.get(username='test_user')
    
    backend/contrib/order/tests/test_tasks.py:53: 
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    venv/lib/python3.9/site-packages/django/db/models/manager.py:85: in manager_method
        return getattr(self.get_queryset(), name)(*args, **kwargs)
    venv/lib/python3.9/site-packages/django/db/models/query.py:431: in get
        num = len(clone)
    venv/lib/python3.9/site-packages/django/db/models/query.py:262: in __len__
        self._fetch_all()
    venv/lib/python3.9/site-packages/django/db/models/query.py:1324: in _fetch_all
        self._result_cache = list(self._iterable_class(self))
    venv/lib/python3.9/site-packages/django/db/models/query.py:51: in __iter__
        results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
    venv/lib/python3.9/site-packages/django/db/models/sql/compiler.py:1173: in execute_sql
        cursor = self.connection.cursor()
    venv/lib/python3.9/site-packages/django/utils/asyncio.py:26: in inner
        return func(*args, **kwargs)
    venv/lib/python3.9/site-packages/django/db/backends/base/base.py:259: in cursor
        return self._cursor()
    venv/lib/python3.9/site-packages/django/db/backends/base/base.py:237: in _cursor
        return self._prepare_cursor(self.create_cursor(name))
    venv/lib/python3.9/site-packages/django/db/utils.py:90: in __exit__
        raise dj_exc_value.with_traceback(traceback) from exc_value
    venv/lib/python3.9/site-packages/django/db/backends/base/base.py:237: in _cursor
        return self._prepare_cursor(self.create_cursor(name))
    venv/lib/python3.9/site-packages/django/utils/asyncio.py:26: in inner
        return func(*args, **kwargs)
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    
    self = <django.contrib.gis.db.backends.postgis.base.DatabaseWrapper object at 0x7fde9030a820>, name = None
    
        @async_unsafe
        def create_cursor(self, name=None):
            if name:
                # In autocommit mode, the cursor will be used outside of a
                # transaction, hence use a holdable cursor.
                cursor = self.connection.cursor(name, scrollable=False, withhold=self.connection.autocommit)
            else:
    >           cursor = self.connection.cursor()
    E           django.db.utils.InterfaceError: connection already closed
    
    venv/lib/python3.9/site-packages/django/db/backends/postgresql/base.py:236: InterfaceError
    ------------------------------------------------------------------------------------- Captured log setup -------------------------------------------------------------------------------------
    ERROR    celery.utils.dispatch.signal:signal.py:280 Signal handler <bound method DjangoWorkerFixup.on_worker_process_init of <celery.fixups.django.DjangoWorkerFixup object at 0x7fde876996a0>> raised: InterfaceError('connection already closed')
    Traceback (most recent call last):
      File "/home/vip/projects/ty_delivery/src/backend/venv/lib/python3.9/site-packages/celery/utils/dispatch/signal.py", line 276, in send
        response = receiver(signal=self, sender=sender, **named)
      File "/home/vip/projects/ty_delivery/src/backend/venv/lib/python3.9/site-packages/celery/fixups/django.py", line 149, in on_worker_process_init
        self._maybe_close_db_fd(c.connection)
      File "/home/vip/projects/ty_delivery/src/backend/venv/lib/python3.9/site-packages/celery/fixups/django.py", line 157, in _maybe_close_db_fd
        _maybe_close_fd(fd)
      File "/home/vip/projects/ty_delivery/src/backend/venv/lib/python3.9/site-packages/celery/fixups/django.py", line 24, in _maybe_close_fd
        os.close(fh.fileno())
    psycopg2.InterfaceError: connection already closed
    ================================================================================== short test summary info ===================================================================================
    FAILED backend/contrib/order/tests/test_tasks.py::TestOrderTask::test_set_profile_orders_total_count - django.db.utils.InterfaceError: connection already closed
    ========================================================================== 1 failed, 1 passed, 15 warnings in 9.49s =========================================================================
    

[sweetpythoncode]

Same problem:

    self = <django.contrib.gis.db.backends.postgis.base.DatabaseWrapper object at 0x7fde9030a820>, name = None
    
        def _cursor(self, name=None):
            self.ensure_connection()
            with self.wrap_database_errors:
    >           return self._prepare_cursor(self.create_cursor(name))
    
    venv/lib/python3.9/site-packages/django/db/backends/base/base.py:237: 
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    
    args = (<django.contrib.gis.db.backends.postgis.base.DatabaseWrapper object at 0x7fde9030a820>, None), kwargs = {}
    event_loop = <_UnixSelectorEventLoop running=False closed=False debug=False>
    
        @functools.wraps(func)
        def inner(*args, **kwargs):
            if not os.environ.get('DJANGO_ALLOW_ASYNC_UNSAFE'):
                # Detect a running event loop in this thread.
                try:
                    event_loop = asyncio.get_event_loop()
                except RuntimeError:
                    pass
                else:
                    if event_loop.is_running():
                        raise SynchronousOnlyOperation(message)
            # Pass onwards.
    >       return func(*args, **kwargs)
    
    venv/lib/python3.9/site-packages/django/utils/asyncio.py:26: 
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    
    self = <django.contrib.gis.db.backends.postgis.base.DatabaseWrapper object at 0x7fde9030a820>, name = None
    
        @async_unsafe
        def create_cursor(self, name=None):
            if name:
                # In autocommit mode, the cursor will be used outside of a
                # transaction, hence use a holdable cursor.
                cursor = self.connection.cursor(name, scrollable=False, withhold=self.connection.autocommit)
            else:
    >           cursor = self.connection.cursor()
    E           psycopg2.InterfaceError: connection already closed
    
    venv/lib/python3.9/site-packages/django/db/backends/postgresql/base.py:236: InterfaceError
    
    The above exception was the direct cause of the following exception:
    
    self = <backend.contrib.order.tests.test_tasks.TestOrderTask object at 0x7fde878a3b50>, celery_worker = <Worker: gen138507@vip-virtual-machine (running)>
    
        def test_set_profile_orders_total_count(self, celery_worker):
    >       profile = UserModel.objects.get(username='test_user')
    
    backend/contrib/order/tests/test_tasks.py:53: 
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    venv/lib/python3.9/site-packages/django/db/models/manager.py:85: in manager_method
        return getattr(self.get_queryset(), name)(*args, **kwargs)
    venv/lib/python3.9/site-packages/django/db/models/query.py:431: in get
        num = len(clone)
    venv/lib/python3.9/site-packages/django/db/models/query.py:262: in __len__
        self._fetch_all()
    venv/lib/python3.9/site-packages/django/db/models/query.py:1324: in _fetch_all
        self._result_cache = list(self._iterable_class(self))
    venv/lib/python3.9/site-packages/django/db/models/query.py:51: in __iter__
        results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
    venv/lib/python3.9/site-packages/django/db/models/sql/compiler.py:1173: in execute_sql
        cursor = self.connection.cursor()
    venv/lib/python3.9/site-packages/django/utils/asyncio.py:26: in inner
        return func(*args, **kwargs)
    venv/lib/python3.9/site-packages/django/db/backends/base/base.py:259: in cursor
        return self._cursor()
    venv/lib/python3.9/site-packages/django/db/backends/base/base.py:237: in _cursor
        return self._prepare_cursor(self.create_cursor(name))
    venv/lib/python3.9/site-packages/django/db/utils.py:90: in __exit__
        raise dj_exc_value.with_traceback(traceback) from exc_value
    venv/lib/python3.9/site-packages/django/db/backends/base/base.py:237: in _cursor
        return self._prepare_cursor(self.create_cursor(name))
    venv/lib/python3.9/site-packages/django/utils/asyncio.py:26: in inner
        return func(*args, **kwargs)
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    
    self = <django.contrib.gis.db.backends.postgis.base.DatabaseWrapper object at 0x7fde9030a820>, name = None
    
        @async_unsafe
        def create_cursor(self, name=None):
            if name:
                # In autocommit mode, the cursor will be used outside of a
                # transaction, hence use a holdable cursor.
                cursor = self.connection.cursor(name, scrollable=False, withhold=self.connection.autocommit)
            else:
    >           cursor = self.connection.cursor()
    E           django.db.utils.InterfaceError: connection already closed
    
    venv/lib/python3.9/site-packages/django/db/backends/postgresql/base.py:236: InterfaceError
    ------------------------------------------------------------------------------------- Captured log setup -------------------------------------------------------------------------------------
    ERROR    celery.utils.dispatch.signal:signal.py:280 Signal handler <bound method DjangoWorkerFixup.on_worker_process_init of <celery.fixups.django.DjangoWorkerFixup object at 0x7fde876996a0>> raised: InterfaceError('connection already closed')
    Traceback (most recent call last):
      File "/home/vip/projects/ty_delivery/src/backend/venv/lib/python3.9/site-packages/celery/utils/dispatch/signal.py", line 276, in send
        response = receiver(signal=self, sender=sender, **named)
      File "/home/vip/projects/ty_delivery/src/backend/venv/lib/python3.9/site-packages/celery/fixups/django.py", line 149, in on_worker_process_init
        self._maybe_close_db_fd(c.connection)
      File "/home/vip/projects/ty_delivery/src/backend/venv/lib/python3.9/site-packages/celery/fixups/django.py", line 157, in _maybe_close_db_fd
        _maybe_close_fd(fd)
      File "/home/vip/projects/ty_delivery/src/backend/venv/lib/python3.9/site-packages/celery/fixups/django.py", line 24, in _maybe_close_fd
        os.close(fh.fileno())
    psycopg2.InterfaceError: connection already closed
    ================================================================================== short test summary info ===================================================================================
    FAILED backend/contrib/order/tests/test_tasks.py::TestOrderTask::test_set_profile_orders_total_count - django.db.utils.InterfaceError: connection already closed
    ========================================================================== 1 failed, 1 passed, 15 warnings in 9.49s =========================================================================

My problem because of transaction.atomic() in @pytest.fixture(scope='session') fixture, mb its help u too.

[Waszker]

Hi, I've got the same issue. It turns out that using fixture with wider scope than function (in my case it is module scope) that has the transaction.atomic() statement, breaks the tests.

Is there any fix to that? Removing db transaction is out of question for me, I need to revert changes made to the database after all the tests in the module.

Or am I understanding it wrong? Are the populated objects removed from the database when the fixture scope "ends"?

[aimestereo]

Confirm this bug

versions:

pytest==6.2.5

# works
pytest-django==4.2.0

# fails
pytest-django==4.3.0 | 4.4.0

in our case we are using class scope fixture to set DB 1 time for all class tests:

@pytest.fixture(scope="class")
def activate_db(request, django_db_setup, django_db_blocker):
    django_db_blocker.unblock()
    tran = django.db.transaction.atomic()
    tran.__enter__()

    yield

    def teardown():
        tran.__exit__(Exception(), None, None)
        django_db_blocker.restore()

    request.addfinalizer(teardown)

minimal setup:

class BaseTest:
    @pytest.fixture(autouse=True, scope="class")
    def prepare(self, activate_db):
        User.objects.count()

    def test_1(self):
        User.objects.count()

    def test_2(self):
        # this test will fail for TestFail
        User.objects.count()


@pytest.mark.django_db  # this mark is the only difference between TestFail and TestPasses
class TestFail(BaseTest):
    pass


class TestPasses(BaseTest):
    pass

---

P.S. so, in my tests I've just removed unnecessary @pytest.mark.django_db and tests started working as expected

[Waszker]

Just a heads-up that the newest version 4.5.2 solved the issue for me! I encourage you to try it out.

[intgr]

I can confirm that the issue no longer occurs with pytest-django 4.5.2, thank you!

[bluetech]

Closing based on latest comments.