synapse
synapse copied to clipboard
element-hq
retry device resync doesn't follow exponentially back off algorithm
This issue has been migrated from #9603.
Description
As far as I'm aware matrix-synapse should stop syncing to dead servers using an exponential back off algorithm. Reference: https://github.com/matrix-org/synapse/issues/4979 https://github.com/matrix-org/synapse/issues/5113
This actually doesn't happen if the error is "No Content-Type header received from remote server". Most likely the server owner disabled the server and left the reverse proxy online or the domain got resold and now there is another HTTP/HTTPS service on that domain.
In my case my server tries to contact it almost twice a minute and is spamming my logs. The exponential back off algorithm does not work in that case.
Logs in my case:
`2021-03-12 12:12:11,582 - synapse.http.matrixfederationclient - 203 - WARNING - _maybe_retry_device_resync-1834 - {GET-O-15116} [PRIVATE.me] Error reading response GET matrix://PRIVATE.me/_matrix/federation/v1/user/devices/%40PRIVATE%3APRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:12:11,583 - synapse.handlers.device - 866 - WARNING - _maybe_retry_device_resync-1834 - Failed to handle device list update for @PRIVATE:PRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:12:40,812 - synapse.http.matrixfederationclient - 203 - WARNING - _maybe_retry_device_resync-1835 - {GET-O-15120} [PRIVATE.me] Error reading response GET matrix://PRIVATE.me/_matrix/federation/v1/user/devices/%40PRIVATE%3APRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:12:40,813 - synapse.handlers.device - 866 - WARNING - _maybe_retry_device_resync-1835 - Failed to handle device list update for @PRIVATE:PRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:13:10,854 - synapse.http.matrixfederationclient - 203 - WARNING - _maybe_retry_device_resync-1836 - {GET-O-15126} [PRIVATE.me] Error reading response GET matrix://PRIVATE.me/_matrix/federation/v1/user/devices/%40PRIVATE%3APRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:13:10,854 - synapse.handlers.device - 866 - WARNING - _maybe_retry_device_resync-1836 - Failed to handle device list update for @PRIVATE:PRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:13:40,815 - synapse.http.matrixfederationclient - 203 - WARNING - _maybe_retry_device_resync-1837 - {GET-O-15130} [PRIVATE.me] Error reading response GET matrix://PRIVATE.me/_matrix/federation/v1/user/devices/%40PRIVATE%3APRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:13:40,815 - synapse.handlers.device - 866 - WARNING - _maybe_retry_device_resync-1837 - Failed to handle device list update for @PRIVATE:PRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:14:10,813 - synapse.http.matrixfederationclient - 203 - WARNING - _maybe_retry_device_resync-1838 - {GET-O-15134} [PRIVATE.me] Error reading response GET matrix://PRIVATE.me/_matrix/federation/v1/user/devices/%40PRIVATE%3APRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:14:10,814 - synapse.handlers.device - 866 - WARNING - _maybe_retry_device_resync-1838 - Failed to handle device list update for @PRIVATE:PRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:14:40,814 - synapse.http.matrixfederationclient - 203 - WARNING - _maybe_retry_device_resync-1839 - {GET-O-15138} [PRIVATE.me] Error reading response GET matrix://PRIVATE.me/_matrix/federation/v1/user/devices/%40PRIVATE%3APRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:14:40,815 - synapse.handlers.device - 866 - WARNING - _maybe_retry_device_resync-1839 - Failed to handle device list update for @PRIVATE:PRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:15:10,814 - synapse.http.matrixfederationclient - 203 - WARNING - _maybe_retry_device_resync-1840 - {GET-O-15142} [PRIVATE.me] Error reading response GET matrix://PRIVATE.me/_matrix/federation/v1/user/devices/%40PRIVATE%3APRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server`
`2021-03-12 12:15:10,814 - synapse.handlers.device - 866 - WARNING - _maybe_retry_device_resync-1840 - Failed to handle device list update for @PRIVATE:PRIVATE.me: Failed to send request: RuntimeError: No Content-Type header received from remote server
Steps to reproduce
- install synapse
- find a user whose server is returning no content-type header
- check logs
Version information
- Homeserver:
-
Version: 1.29.0+buster1
-
Install method: Debian Package Manager
- Platform: Debian, LXC Container
My logfiles are filled with messages like
2024-09-10 08:07:42,972 - synapse.handlers.device - 1501 - WARNING - _maybe_retry_device_resync-19 - Failed to handle device list update for @orloskynedi:nyctico.de: 301: Moved Permanently
2024-09-10 08:07:42,994 - synapse.http.matrixfederationclient - 858 - WARNING - _maybe_retry_device_resync-19 - {GET-O-5059} [chat.paralilo.net] Request failed: GET matrix-federation://chat.paralilo.net/_matrix/federation/v1/user/devices/%40feliciaabebi%3Achat.paralilo.net: HttpResponseException('404: Not Found')
2024-09-10 08:07:42,994 - synapse.handlers.device - 1501 - WARNING - _maybe_retry_device_resync-19 - Failed to handle device list update for @feliciaabebi:chat.paralilo.net: 404: Not Found
2024-09-10 08:07:43,014 - synapse.http.matrixfederationclient - 858 - WARNING - _maybe_retry_device_resync-19 - {GET-O-5060} [nyctico.de] Request failed: GET matrix-federation://nyctico.de/_matrix/federation/v1/user/devices/%40glanville_luisa%3Anyctico.de: HttpResponseException('301: Moved Permanently')
2024-09-10 08:07:43,014 - synapse.handlers.device - 1501 - WARNING - _maybe_retry_device_resync-19 - Failed to handle device list update for @glanville_luisa:nyctico.de: 301: Moved Permanently
2024-09-10 08:07:43,037 - synapse.http.matrixfederationclient - 858 - WARNING - _maybe_retry_device_resync-19 - {GET-O-5061} [chat.paralilo.net] Request failed: GET matrix-federation://chat.paralilo.net/_matrix/federation/v1/user/devices/%40thadeusnepean%3Achat.paralilo.net: HttpResponseException('404: Not Found')
2024-09-10 08:07:43,037 - synapse.handlers.device - 1501 - WARNING - _maybe_retry_device_resync-19 - Failed to handle device list update for @thadeusnepean:chat.paralilo.net: 404: Not Found
2024-09-10 08:07:43,060 - synapse.http.matrixfederationclient - 858 - WARNING - _maybe_retry_device_resync-19 - {GET-O-5062} [nyctico.de] Request failed: GET matrix-federation://nyctico.de/_matrix/federation/v1/user/devices/%40grider_kellsie%3Anyctico.de: HttpResponseException('301: Moved Permanently')
2024-09-10 08:07:43,061 - synapse.handlers.device - 1501 - WARNING - _maybe_retry_device_resync-19 - Failed to handle device list update for @grider_kellsie:nyctico.de: 301: Moved Permanently
2024-09-10 08:07:43,082 - synapse.http.matrixfederationclient - 858 - WARNING - _maybe_retry_device_resync-19 - {GET-O-5063} [nyctico.de] Request failed: GET matrix-federation://nyctico.de/_matrix/federation/v1/user/devices/%40kaceygabriello%3Anyctico.de: HttpResponseException('301: Moved Permanently')
It starts every 20 to 30 seconds.
Version: matrix-synapse-py3 1.114.0+bookworm1
As an admin is there anything I can do manually to make Synapse forget these and stop trying?
I also have a problem with _maybe_retry_device_resync twice a minute
synapse.handlers.device - 1786 - INFO - _maybe_retry_device_resync-1277- 403: Federation denied with halogen.place.
