Frequent httpx.RemoteProtocolError: Server disconnected

[immortal3]

Bug report

• [x] I confirm this is a bug with Supabase, not with my own application.
• [x] I confirm I have searched the Docs, GitHub Discussions, and Discord.

Describe the bug

When making API requests to Supabase using PostgREST client, the server unexpectedly disconnects, resulting in a httpx.RemoteProtocolError: Server disconnected error. This happens intermittently when trying to retrieve data from a specific table or batch insert.

To Reproduce

Steps to reproduce the behavior:

1. Set up a connection to Supabase using client
2. Attempt to execute a query to retrieve data from a table
3. The server disconnects during the request, throwing a RemoteProtocolError

Code snippet demonstrating the issue:

# Using postgrest client to query a table
result = client.table("my_table").select("*").eq("key", "value").execute()
# This results in server disconnection

Expected behavior

The query should complete successfully and return the requested data without any server disconnection.

System information

• OS: Linux
• Version of postgrest-py: [latest]
• Version of httpx: [latest]
• Python version: 3.11

Additional context

As of now, We have added retry mechanism over certain call during exception of HTTPX disconnect. Following stack trace,

  File "/usr/local/lib/python3.11/site-packages/postgrest/_sync/request_builder.py", line 58, in execute
    r = self.session.request(
        ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 825, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/dd_tracer/python/ddtrace/contrib/internal/httpx/patch.py", line 166, in _wrapped_sync_send
    resp = wrapped(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1014, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 249, in handle_request
    with map_httpcore_exceptions():
  File "/usr/local/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.RemoteProtocolError: Server disconnected

[linear[bot]]

CLIBS-116 Frequent httpx.RemoteProtocolError: Server disconnected

[silentworks]

The reproduction steps you provide wouldn't yield much for anyone since there is no data with the steps. I use the Python library on a regular basis and haven't experienced this issue. It could be from a query taking too long and causing a timeout then the client disconnects. It could be that your network lost connection at some point for the client to disconnect. You should at least create a reproducible repo for something like this as it will be hard to reproduce otherwise.

[immortal3]

@silentworks I agree it's challenging to reproduce this issue, but ideally the Supabase team would have documentation that points to where to look for this specific bug. Whether it's a server timeout (which should produce a distinct error message) or a lost network connection, this appears to be a recurring problem that's impacting our production environment. Having clearer troubleshooting guidance would be extremely helpful for resolving these frequent issues.

[silentworks]

This is definitely not how error reporting and bug fixing works. You should first provide a reproduction steps that should result in the issue resurfacing, without this how will someone fix the issue and make sure that the fix works? server timeout doesn't produce a distinct error nor does network connection lost. If you believe this is an Supabase infrastructure issue then you should open a support request on Supabase's official support channel https://supabase.com/dashboard/support/new

[immortal3]

@silentworks i have already opened support ticket, also, i had wrong description of bug. it is intermittently and not consistent, so 100% (even 10%) re-produceable code is not possible. We are facing this in 3-4 times in a day.

[silentworks]

I'm going to look into this to see if I can recreate this using local Supabase. Let's keep this issue open so I can note my findings here.

[loner233]

I also encountered the same issue. I am using Python 3.11 and the latest supabase-py. I have a cron job in Kubernetes that periodically fetches some data. It works fine 99% of the time, but occasionally, this error occurs.

[vasanth-asokan]

@silentworks in case it helps, we don't see corresponding "disconnection" errors or anything anomalous in the Supabase logs (we've checked API gateway logs and Database logs). This is now frequently happening on another Supabase DB we use.

We did upgrade to the latest supabase-py (2.13.0) a couple of weeks ago. Prior to that we had been on 2.11.0.

[analyst3dev]

Also encountered the same issue. With intermittent disconnection issues that are consistently triggering lately and can be isolated to supabase api calls. Also no logs/errors in Supabase logs. Using supabase python client (2.4.0).

[DDoerner]

Can confirm that this issue also affects the storage API where we had this issue be thrown occassionally by either SyncBucketProxy.exists() or SyncBucketProxy.upload().

[DDoerner]

The Supabase support pointed us to the following issues raised with the underlying httpx package, most likely caused by issues with keeping the HTTP/2 connection alive for an extended period.

https://github.com/encode/httpx/issues/3324 https://github.com/encode/httpx/issues/1862 https://github.com/encode/httpx/discussions/2056

For now we've solved this problem with a generic retrying wrapper for all Supabase calls.

[vasanth-asokan]

@DDoerner glad you heard these pointers from Supabase support. We were being pointed to unrelated 406 errors with a separate node client, which had nothing to do with the errors we are seeing from supabase-py.

The pointers to the httpx issues you've gotten seem closer to the problem we see on our end as well. While we also mitigated things by introducing retries and but seem it happening quite often on a given day (dozens of times across 1000s of requests). Would you be able to share what sort of retry approach seems to work for you? We have a 10 second backoff and retry (2x) and this happens often enough to make working with supabase-py unreliable.

[silentworks]

We're currently working on allowing you to pass your own httpx client as the client itself allows you to set retries through it's HTTPTransport class. I also found that setting max_keepalive_connections to 1 instead of the default of 20 has worked better in my tests. You can find the test httpx setup I've done here https://github.com/silentworks/sb-python-issues/blob/main/supabase_py_1075/main.py#L30-L56, but soon you will be able to pass the httpx client itself to supabase-py so you can change the default configurations. You can read more on this issue https://github.com/supabase/supabase-py/issues/1075.

[DDoerner]

@vasanth-asokan We haven't really found a better strategy for the time being and use a limited retry count and exponential backoff. That seems good enough for now, but it isn't the best solution long-term.

@silentworks that sounds good, we'll try it out.

[silentworks]

As of 2.16.0, you can now pass your own httpx client to the library. You can configure different settings and see which works best in your server setup.

from httpx import Client, HTTPTransport, Limits
from supabase import create_client, ClientOptions

transport = HTTPTransport(
    retries=3,
    http2=True,
    limits=Limits(
        max_connections=100,
        max_keepalive_connections=1,
        keepalive_expiry=None,
    )
)

with Client(transport=transport) as http_client:
    # Create a client with the custom httpx client
    options = ClientOptions(httpx_client=http_client)

    client = create_client(supabase_url, supabase_key, options=options)

[Ousret]

If you are ever interested, as httpx is stale for over a year now regarding those issues. You may be interested in looking at Niquests, this one will not cause those spurious errors you've all seen.

regards,

[silentworks]

@Ousret you should state that you are associated with the project you are suggesting here. This is the first I'm hearing of it and httpx is more widely known and used. Maybe when Niquests becomes more widely used then we will take a look at it.

[Ousret]

Duly noted. Sorry.

Yes httpx is "more widely used and known" but its current state is a bit scary if you've followed major subjects over there. It have major flaws unpatched for years. (eg. search for httpx + ProtocolError on GH alone will yield hundred of results/complaints across many projects)

At least now, you know about it. That's a start.

[silentworks]

Closing this out as a solution was provided above.