Compatibility status with Psycopg v3

[cro]

Greetings,

I have an existing Django project I am retrofitting to use Channels and channels_postgres with, and it uses Psycopg version 3.

I had a toy project I started from scratch to experiment with Channels and channels_postgres. The toy project worked fine, but the retrofitted project was throwing tracebacks (see below) in the channels layers. django-admin startproject defaults to v3.

I can make the tracebacks go away if I remove Psycopg v3 and replace it with v2.

I see the reference to django.db.backends.postgresql_psycopg2 in the documentation. Docs for recent versions of Django seem to imply that you can use django.db.backends.postgresql and Django will do the "right thing" with respect to the available database driver.

Is this a known issue? Even though I explicitly replaced the channels ENGINE with django.db.backends.postgresql_psycopg2 the Django database subsystem seems to ignore it and load v3.

I took a quick stab at attempting to remove aiopg and replace it with Psycopg 3 since version 3 supports async now, but it is proving to be more effort than I thought. I also was unsure if you would accept a PR because of potential compatibility mismatches across the version matrix.

Traceback (most recent call last):
  File "/Users/cro/src/odmn/.venv/lib/python3.13/site-packages/django/core/handlers/exception.py", line 42, in inner
    response = await get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cro/src/odmn/.venv/lib/python3.13/site-packages/django/core/handlers/base.py", line 253, in _get_response_async
    response = await wrapped_callback(
               
  File "/Users/cro/.local/share/uv/python/cpython-3.13.2-macos-aarch64-none/lib/python3.13/asyncio/futures.py", line 286, in __await__
    yield self  # This tells Task to wait for completion.
    ^^^^^^^^^^
  File "/Users/cro/.local/share/uv/python/cpython-3.13.2-macos-aarch64-none/lib/python3.13/asyncio/tasks.py", line 375, in __wakeup
    future.result()
    ^^^^^^^^^^^^^^^
  File "/Users/cro/.local/share/uv/python/cpython-3.13.2-macos-aarch64-none/lib/python3.13/asyncio/futures.py", line 199, in result
    raise self._exception.with_traceback(self._exception_tb)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cro/src/odmn/odmn/core/views.py", line 173, in eventtest
    async_to_sync(channel_layer.group_send)("chat",
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cro/.local/share/uv/python/cpython-3.13.2-macos-aarch64-none/lib/python3.13/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/cro/.local/share/uv/python/cpython-3.13.2-macos-aarch64-none/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
    ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cro/src/odmn/.venv/lib/python3.13/site-packages/channels_postgres/core.py", line 266, in group_send
    _, pool = await self.get_pool()
              ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cro/src/odmn/.venv/lib/python3.13/site-packages/channels_postgres/core.py", line 76, in get_pool
    pool = await aiopg.create_pool(**self.db_params, **self.async_lib_config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cro/src/odmn/.venv/lib/python3.13/site-packages/aiopg/pool.py", line 300, in from_pool_fill
    await self._fill_free_pool(False)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cro/src/odmn/.venv/lib/python3.13/site-packages/aiopg/pool.py", line 336, in _fill_free_pool
    conn = await connect(
                 
  File "/Users/cro/src/odmn/.venv/lib/python3.13/site-packages/aiopg/connection.py", line 65, in connect
    connection = Connection(
                 
  File "/Users/cro/src/odmn/.venv/lib/python3.13/site-packages/aiopg/connection.py", line 760, in __init__
    self._conn = psycopg2.connect(dsn, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cro/src/odmn/.venv/lib/python3.13/site-packages/psycopg2/__init__.py", line 121, in connect
    dsn = _ext.make_dsn(dsn, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cro/src/odmn/.venv/lib/python3.13/site-packages/psycopg2/extensions.py", line 167, in make_dsn
    parse_dsn(dsn)
    ^^^^^^^^^^^^^^

Exception Type: ProgrammingError at /core/eventtest/
Exception Value: invalid dsn: invalid connection option "context"```

[wolph]

I'm not sure what the impact of it would be, but it's fixable by adding an extra line here:

https://github.com/danidee10/channels_postgres/blob/8ac3fd69449ee2f0a4bcb38189365e85d123d745/channels_postgres/core.py#L65

It needs something like this:

self.db_params.pop('context', None)

It's caused by psycopg3 passing the timezone in the context.

[wolph]

I should note that the proper fix is a bit more involved. The library currently requires aiopg which is a deprecated psycopg2 wrapper. With a proper fix the aiopg dependency would be dropped since psycopg v3 properly supports asyncio native.

https://github.com/danidee10/channels_postgres/blob/8ac3fd69449ee2f0a4bcb38189365e85d123d745/channels_postgres/core.py#L68-L80

[danidee10]

@wolph Yeah. A proper fix will Involve replacing aiopg.

But It's actually going in the right direction; because the reason I used aiopg at the time was the lack of asyncio in psycopg2

[danidee10]

Greetings,

I have an existing Django project I am retrofitting to use Channels and channels_postgres with, and it uses Psycopg version 3.

I had a toy project I started from scratch to experiment with Channels and channels_postgres. The toy project worked fine, but the retrofitted project was throwing tracebacks (see below) in the channels layers. django-admin startproject defaults to v3.

I can make the tracebacks go away if I remove Psycopg v3 and replace it with v2.

I see the reference to django.db.backends.postgresql_psycopg2 in the documentation. Docs for recent versions of Django seem to imply that you can use django.db.backends.postgresql and Django will do the "right thing" with respect to the available database driver.

Is this a known issue? Even though I explicitly replaced the channels ENGINE with django.db.backends.postgresql_psycopg2 the Django database subsystem seems to ignore it and load v3.

I took a quick stab at attempting to remove aiopg and replace it with Psycopg 3 since version 3 supports async now, but it is proving to be more effort than I thought. I also was unsure if you would accept a PR because of potential compatibility mismatches across the version matrix.

Traceback (most recent call last):
  File "/Users/cro/src/odmn/.venv/lib/python3.13/site-packages/django/core/handlers/exception.py", line 42, in inner
    response = await get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cro/src/odmn/.venv/lib/python3.13/site-packages/django/core/handlers/base.py", line 253, in _get_response_async
    response = await wrapped_callback(
               
  File "/Users/cro/.local/share/uv/python/cpython-3.13.2-macos-aarch64-none/lib/python3.13/asyncio/futures.py", line 286, in __await__
    yield self  # This tells Task to wait for completion.
    ^^^^^^^^^^
  File "/Users/cro/.local/share/uv/python/cpython-3.13.2-macos-aarch64-none/lib/python3.13/asyncio/tasks.py", line 375, in __wakeup
    future.result()
    ^^^^^^^^^^^^^^^
  File "/Users/cro/.local/share/uv/python/cpython-3.13.2-macos-aarch64-none/lib/python3.13/asyncio/futures.py", line 199, in result
    raise self._exception.with_traceback(self._exception_tb)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cro/src/odmn/odmn/core/views.py", line 173, in eventtest
    async_to_sync(channel_layer.group_send)("chat",
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cro/.local/share/uv/python/cpython-3.13.2-macos-aarch64-none/lib/python3.13/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/cro/.local/share/uv/python/cpython-3.13.2-macos-aarch64-none/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
    ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cro/src/odmn/.venv/lib/python3.13/site-packages/channels_postgres/core.py", line 266, in group_send
    _, pool = await self.get_pool()
              ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cro/src/odmn/.venv/lib/python3.13/site-packages/channels_postgres/core.py", line 76, in get_pool
    pool = await aiopg.create_pool(**self.db_params, **self.async_lib_config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cro/src/odmn/.venv/lib/python3.13/site-packages/aiopg/pool.py", line 300, in from_pool_fill
    await self._fill_free_pool(False)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cro/src/odmn/.venv/lib/python3.13/site-packages/aiopg/pool.py", line 336, in _fill_free_pool
    conn = await connect(
                 
  File "/Users/cro/src/odmn/.venv/lib/python3.13/site-packages/aiopg/connection.py", line 65, in connect
    connection = Connection(
                 
  File "/Users/cro/src/odmn/.venv/lib/python3.13/site-packages/aiopg/connection.py", line 760, in __init__
    self._conn = psycopg2.connect(dsn, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cro/src/odmn/.venv/lib/python3.13/site-packages/psycopg2/__init__.py", line 121, in connect
    dsn = _ext.make_dsn(dsn, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/cro/src/odmn/.venv/lib/python3.13/site-packages/psycopg2/extensions.py", line 167, in make_dsn
    parse_dsn(dsn)
    ^^^^^^^^^^^^^^

Exception Type: ProgrammingError at /core/eventtest/
Exception Value: invalid dsn: invalid connection option "context"```

It should be possible to replace aiopg totally without any breaking changes to the public interface. I just checked and the latest stable version of psycopg3 (3.2.6 at the time of writing) supports Python >=3.8 https://github.com/psycopg/psycopg/blob/3.2.6/psycopg/setup.cfg#L54

[danidee10]

I have a WIP branch for pyscopg3: https://github.com/danidee10/channels_postgres/pull/38

I replaced aiopg with pyscopg3 and it appears to be working fine so far. You can give it a shot and see if it works for you

[cro]

Fantastic! I will try this out!

[wolph]

I've done limited testing so far, but I'm not seeing any issues yet. Great work, thank you!

[danidee10]

version 1.1.1 has been released on Pypi! https://pypi.org/project/channels-postgres/

While I was at it, I also made a lot of performance Improvements. One of them being the number of concurrent clients that the layer could handle.

Previously, the maximum amount of clients was essentially the same as the amount of open connections that the Postgres server could handle

The performance isn't as good as the official Redis layer (in terms of throughput) due to the general overhead of a general-purpose RDBMS vs an in-memory key-value store but now, it should be able to handle hundreds of connected clients with small to medium amounts of traffic.

Please try it out and let me know if it works for you.

[wolph]

I've been trying it a bit and it seems to be replicating the messages depending on the number of connections. If there are 2 database connections it's executing it 2 times, if you have 4 connections it will execute the function (when an event arrives from a websocket) 4 times.

I've verified by downgrading to the old psycopg2 version and it's definitely an issue that's been introduced with the new version. I'm not sure I can tell you how to reliably reproduce it though...

[danidee10]

What do you mean by number of connections ?

Are you running multiple workers (which creates two database connection pools) or do you just mean multiple websocket clients ?

PS: I tried both options and I couldn't reproduce it

[wolph]

My code is currently doing something like this:

1. The browser sends a websocket message to the main consumer (running via runserver)
2. The main consumer sends a channels message to a non-websocket consumer (running via runworker).
3. The non-websocket consumer does some processing and sends a message back to the main consumer.

The main consumer receives a single message from the websocket as far as I can see, but the non-websocket consumer gets 2 or more messages, but they do have a different requests_num.

Yesterday (with the debug logs enabled) I could see that if connections_num was 4, it would send it 4 times. Right now I'm just seeing the messages twice even though there are 4 connections both in the runserver and the runworker.

I can reliably reproduce it with the new version and the old version does not show the behaviour.

[danidee10]

I couldn't reproduce it :(

But I set up a repository with my attempts here.

• 1 Websocket cli client (Browser)
• The WebSocket server running with daphne (Consumer)
• A worker processing messages and sending a response back to the client (non-WebSocket consumer)

It'd be great if you could use that to reproduce the error.

I did find a case where we get dual messages but that only happens when running multiple workers on the same channel. both workers get the same message because there's no lock or indication that one worker is already processing the message.

Which goes against the channels layer spec of "at most one delivery". The previous version didn't have this problem because it used the NOTIFY as a signal and always fetched the message from the database with SELECT...FOR UPDATE which implicitly acquired a database lock.

I removed it (for performance reasons) and read directly from the NOTIFY signal

I think you're experiencing the same bug but through a single consumer ? 🤷

[wolph]

I think I've figured it out... I am running runworker consumer_a consumer_b, that causes it. When it's a single worker there is no issue.

That's at least a good workaround for me, I was only doing that for development purposes.

But they're using different channels:

channels = dict(
    consumer_a=consumers.ConsumerA.as_asgi(),
    consumer_b=consumers.ConsumerB.as_asgi(),
)

application = ProtocolTypeRouter(
    {
        'http': django_asgi_app,
        'websocket': URLRouter(urlpatterns),
        'channel': ChannelNameRouter(channels),
    },
)

[danidee10]

Yep. it was a bug. runworker with multiple channels creates a coroutine for every specified channel.

Essentially behaving like Independent workers and hitting the bug that I mentioned above (with the caveat that asyncio cannot truly process them in parallel like a separate process)

It should be fixed in the latest version 1.1.2!

[wolph]

Thank you for the quick fix, that indeed solves it!

But now I'm seeing the bug the other way around. When the runworker sends a message back to the websocket (using the specific channel), I'm seeing the messages twice.

I don't think that's a new bug, it would explain the 4 times I saw earlier. Duplicating both back and forth.