An error occurred: You can't write against a read only replica

[gasaichandesu]

At some point of time we started to get can't write to read only replica exceptions. Here is what we have in logs:

^[[36mevent-handler_1            |^[[0m 2025-05-06 05:19:57,044 - ERROR - An error occurred: You can't write against a read only replica.
^[[36mevent-handler_1            |^[[0m Traceback (most recent call last):
^[[36mevent-handler_1            |^[[0m   File "/app/event_handler.py", line 320, in handle_subscription
^[[36mevent-handler_1            |^[[0m     await asyncio.gather(*(query_database(key, f) for key, f in cache_misses))
^[[36mevent-handler_1            |^[[0m   File "/app/event_handler.py", line 316, in query_database
^[[36mevent-handler_1            |^[[0m     await redis_client.setex(cache_key, 240, orjson.dumps(parsed_results))
^[[36mevent-handler_1            |^[[0m   File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/redis/__init__.py", line 389, in _async_traced_execute_command
^[[36mevent-handler_1            |^[[0m     response = await func(*args, **kwargs)
^[[36mevent-handler_1            |^[[0m                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
^[[36mevent-handler_1            |^[[0m   File "/usr/local/lib/python3.11/site-packages/redis/asyncio/client.py", line 550, in execute_command
^[[36mevent-handler_1            |^[[0m     return await conn.retry.call_with_retry(
^[[36mevent-handler_1            |^[[0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^[[36mevent-handler_1            |^[[0m   File "/usr/local/lib/python3.11/site-packages/redis/asyncio/retry.py", line 59, in call_with_retry
^[[36mevent-handler_1            |^[[0m     return await do()
^[[36mevent-handler_1            |^[[0m            ^^^^^^^^^^
^[[36mevent-handler_1            |^[[0m   File "/usr/local/lib/python3.11/site-packages/redis/asyncio/client.py", line 524, in _send_command_parse_response
^[[36mevent-handler_1            |^[[0m     return await self.parse_response(conn, command_name, **options)
^[[36mevent-handler_1            |^[[0m            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^[[36mevent-handler_1            |^[[0m   File "/usr/local/lib/python3.11/site-packages/redis/asyncio/client.py", line 571, in parse_response
^[[36mevent-handler_1            |^[[0m     response = await connection.read_response()
^[[36mevent-handler_1            |^[[0m                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^[[36mevent-handler_1            |^[[0m   File "/usr/local/lib/python3.11/site-packages/redis/asyncio/connection.py", line 537, in read_response
^[[36mevent-handler_1            |^[[0m     raise response from None
^[[36mevent-handler_1            |^[[0m redis.exceptions.ReadOnlyError: You can't write against a read only replica.
^[[36mevent-handler_1            |^[[0m INFO:     172.28.0.2:46564 - "POST /subscription HTTP/1.1" 500 Internal Server Error

As a result, relay returns an empty response for all filters and it means that users cannot log in, cannot see their feeds, etc.

I want to find out if it is a nostpy bug or something wrong in our infrastructure. Relay is delpoyed using menu.py so it's a docker-compose stack