element-hq
Inbound federation fails after upgrading to synapse 1.140
Description
Local users no longer receive events over federation, outbound federation remains functional
Steps to reproduce
- Upgrade to 1.140
- Send a message
- Note that it never arrives on other servers unless the server gets downgraded
Homeserver
the-apothecary.club
Synapse Version
v1.140
Installation Method
Other (please mention below)
Database
Single PG, ported years ago, restored months back
Workers
Single process
Platform
NixOS/unstable, using nixos module, i7-7700, 64GB of ram, no container
Configuration
Presence enabled Using Synapse-HTTP-Antispam Very large caches
experimental_features: msc2815_enabled: true
Relevant log output
Unable to deduce which parts of the log *are* the cause.
Lots of instances of "Error attempting to resove state at missing prev_events", "Request failed: GET matrix-federation://", "ERROR - sync_partial_state_room-0 - Background process 'sync_partial_state_room' threw an exception" followed by stacktraces, "synapse.api.errors.FederationError: ERROR 403: We can't get valid state history.", etc...
I have a 2.3GB log file available I could share privately (Last 10M log lines through journalctl, most likely includes logs from before the attempt to upgrade to 1.140).
Anything else that would be useful to know?
No response
Same here (archlinux.org), using antispam with Draupnir. Frequent errors in the logs like this one:
synapse_http_antispam: [PUT-762951-!…] Error in async callback (POST http://localhost:8585/api/1/spam_check/check_event_for_spam)
Traceback (most recent call last):
File "/var/lib/synapse/venv/lib/python3.13/site-packages/synapse_http_antispam.py", line 73, in _catch_errors
await task
File "/var/lib/synapse/venv/lib/python3.13/site-packages/synapse/http/client.py", line 564, in post_json_get_json
response = await self.request(
^^^^^^^^^^^^^^^^^^^
"POST", uri, headers=Headers(actual_headers), data=json_str
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/var/lib/synapse/venv/lib/python3.13/site-packages/synapse/http/client.py", line 448, in request
response = await make_deferred_yieldable(request_deferred)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/synapse/venv/lib/python3.13/site-packages/twisted/internet/defer.py", line 1187, in __iter__
yield self
twisted.web._newclient.RequestGenerationFailed: [<twisted.python.failure.Failure twisted.internet.defer.CancelledError: >]
The module config is:
- module: synapse_http_antispam.HTTPAntispam
config:
base_url: http://localhost:8585/api/1/spam_check
authorization: "…"
enabled_callbacks:
- check_event_for_spam
- user_may_invite
- user_may_join_room
fail_open:
check_event_for_spam: true
user_may_invite: true
user_may_join_room: true
async:
check_event_for_spam: true
for context it seems that disabling check_event_for_spam seems to be a workaround. Not good that safety critical infrastructure has been effected by internal synapse changes though.
