synapse synapse servers from outside DDOSing my server

I dont use synapse anymore, nor matrix. but this has been an issue for the past 2 months and I thought this will solve on its own.

{9BB3A012-5FBE-4E96-9DB0-AD5CB5B5711D}

{4A11DD86-04D1-401D-AAE5-FAA8BBFADED5}

{59C8C07B-CF8F-4A6B-8B6B-5A66D3D6218A}

{8B09054B-C5C1-4BA3-B6C9-F32A8FF44AEE}

Sep 21 '24 06:09 CustomIcon

You shutdown your server and did not leave all rooms.

The rest of the federation is trying to reach your server...

Sep 21 '24 14:09 daedric7

You shutdown your server and did not leave all rooms.

The rest of the federation is trying to reach your server...

does that mean i can still spin up synapse again and leave rooms?

Sep 21 '24 19:09 CustomIcon

{B15DECF1-3AA3-4F7B-9CC4-72558BE5C1DF}

got a solution from a friend. but i dont think it really solves the problem of synapse auto removing my synapse instance after a long period of downtime

Sep 21 '24 21:09 CustomIcon

Looks like #15365, related to #3765. Your webserver is returning a 404 which is #5442

Sep 29 '24 18:09 thisisjoesGH

Just to chime in, after working through some firewall rules and a possible crowdsec solution:

The federation system should not assume that the domain is still owned by the same entity that deployed the node and joined the federation.
I don't understand the "why", what metadata is forcing the federation system to contact a domain after more than a year and not forget/drop it after that long? What is the anchor, a message or join/leave message I produced? I just hope that a new owner could not just take ownership of whatever happened to the domain back then by bringing up a clean instance. On the other hand, what would happen if they were different on all the other servers within the federation?
https://github.com/element-hq/synapse/issues/5442 mentions that 429 status could be a solution, but I could not tell if a Retry-After header would be respected. Is this a viable way to stop (not reduce) requests and mark the server as "destroyed"? I tried to work through the code but had no luck.
Synapse does NOT pass a Referer header. I would not be able to identify the reason why _matrix/* is being called on my target server, something Conduwuit seems to do.

The whole system seems to be very easy to abuse. Within my /.well-known/matrix/server I can point to any target, not limited to the owning ETLD+1, up to the Synapse/1.121.1 user-agent. Just now I could redirect over 500 unique IPs (machines hammering my well-known in the last 4 days), both IPv4 and IPv6, to any destination I wanted. Many of them also RDNSing to consumer-level provider issued domain names.

My server was alive maybe some weeks, joined 4-5 channels/servers(?), over a year ago if I remember correctly.

Here are some insights, grouped by a single (redacted) IP, tested on my own domains with, no /.well-known/matrix/client defined. Router http://127.0.0.1:9021 is my domain receiving the /.well-known/matrix/server requests, http://127.0.0.1:8870 is the target server, logs are NGINX style, grepped directly from the reverse proxy handling both.

`Synapse/1.120.2`

Triggers 2 requests if the target responds with 404 status within 10s. Retrying every ~6m.

REDACTED - - [02/Jan/2025:04:26:22 +0000] "GET /.well-known/matrix/server HTTP/1.1" 200 40 "-" "Synapse/1.120.2" 2735 "REDACTED" "http://127.0.0.1:9021" 1ms
REDACTED - - [02/Jan/2025:04:26:23 +0000] "GET /_matrix/key/v2/server HTTP/1.1" 404 5755 "-" "Synapse/1.120.2" 2736 "REDACTED" "http://127.0.0.1:8870" 16ms
REDACTED - - [02/Jan/2025:04:26:32 +0000] "GET /_matrix/key/v2/server HTTP/1.1" 404 5755 "-" "Synapse/1.120.2" 2738 "REDACTED" "http://127.0.0.1:8870" 2ms

`Synapse/1.118.0`

Triggers 8 requests if the target responds with 404, within 3.5m.

REDACTED - - [02/Jan/2025:03:45:37 +0000] "GET /.well-known/matrix/server HTTP/1.1" 200 40 "-" "Synapse/1.118.0" 2577 "REDACTED" "http://127.0.0.1:9021" 1ms
REDACTED - - [02/Jan/2025:03:45:37 +0000] "GET /_matrix/federation/v1/user/devices/REDACTED HTTP/1.1" 404 5755 "-" "Synapse/1.118.0" 2578 "REDACTED" "http://127.0.0.1:8870" 16ms
REDACTED - - [02/Jan/2025:03:46:06 +0000] "GET /_matrix/federation/v1/user/devices/REDACTED HTTP/1.1" 404 5755 "-" "Synapse/1.118.0" 2579 "REDACTED" "http://127.0.0.1:8870" 2ms
REDACTED - - [02/Jan/2025:03:46:36 +0000] "GET /_matrix/federation/v1/user/devices/REDACTED HTTP/1.1" 404 5755 "-" "Synapse/1.118.0" 2581 "REDACTED" "http://127.0.0.1:8870" 2ms
REDACTED - - [02/Jan/2025:03:47:06 +0000] "GET /_matrix/federation/v1/user/devices/REDACTED HTTP/1.1" 404 5755 "-" "Synapse/1.118.0" 2582 "REDACTED" "http://127.0.0.1:8870" 16ms
REDACTED - - [02/Jan/2025:03:47:36 +0000] "GET /_matrix/federation/v1/user/devices/REDACTED HTTP/1.1" 404 5755 "-" "Synapse/1.118.0" 2586 "REDACTED" "http://127.0.0.1:8870" 2ms
REDACTED - - [02/Jan/2025:03:48:06 +0000] "GET /_matrix/federation/v1/user/devices/REDACTED HTTP/1.1" 404 5755 "-" "Synapse/1.118.0" 2588 "REDACTED" "http://127.0.0.1:8870" 15ms
REDACTED - - [02/Jan/2025:03:48:36 +0000] "GET /_matrix/federation/v1/user/devices/REDACTED HTTP/1.1" 404 5755 "-" "Synapse/1.118.0" 2590 "REDACTED" "http://127.0.0.1:8870" 2ms
REDACTED - - [02/Jan/2025:03:48:55 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/1.1" 404 5755 "-" "Synapse/1.118.0" 2591 "REDACTED" "http://127.0.0.1:8870" 194ms

`Synapse/1.90.0`

Triggers 18 requests if the target responds with 404, within < 10s.

REDACTED - - [02/Jan/2025:04:20:35 +0000] "GET /.well-known/matrix/server HTTP/1.1" 200 40 "-" "Synapse/1.90.0" 2703 "REDACTED" "http://127.0.0.1:9021" 1ms
REDACTED - - [02/Jan/2025:04:20:35 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/1.1" 404 5755 "-" "Synapse/1.90.0" 2704 "REDACTED" "http://127.0.0.1:8870" 16ms
REDACTED - - [02/Jan/2025:04:20:35 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/1.1" 404 5755 "-" "Synapse/1.90.0" 2705 "REDACTED" "http://127.0.0.1:8870" 2ms
REDACTED - - [02/Jan/2025:04:20:36 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/1.1" 404 5755 "-" "Synapse/1.90.0" 2706 "REDACTED" "http://127.0.0.1:8870" 2ms
REDACTED - - [02/Jan/2025:04:20:36 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/1.1" 404 5755 "-" "Synapse/1.90.0" 2707 "REDACTED" "http://127.0.0.1:8870" 2ms
REDACTED - - [02/Jan/2025:04:20:37 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/1.1" 404 5755 "-" "Synapse/1.90.0" 2708 "REDACTED" "http://127.0.0.1:8870" 2ms
REDACTED - - [02/Jan/2025:04:20:37 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/1.1" 404 5755 "-" "Synapse/1.90.0" 2709 "REDACTED" "http://127.0.0.1:8870" 2ms
REDACTED - - [02/Jan/2025:04:20:37 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/1.1" 404 5755 "-" "Synapse/1.90.0" 2710 "REDACTED" "http://127.0.0.1:8870" 2ms
REDACTED - - [02/Jan/2025:04:20:38 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/1.1" 404 5755 "-" "Synapse/1.90.0" 2711 "REDACTED" "http://127.0.0.1:8870" 2ms
REDACTED - - [02/Jan/2025:04:20:38 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/1.1" 404 5755 "-" "Synapse/1.90.0" 2712 "REDACTED" "http://127.0.0.1:8870" 2ms
REDACTED - - [02/Jan/2025:04:20:38 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/1.1" 404 5755 "-" "Synapse/1.90.0" 2713 "REDACTED" "http://127.0.0.1:8870" 2ms
REDACTED - - [02/Jan/2025:04:20:39 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/1.1" 404 5755 "-" "Synapse/1.90.0" 2714 "REDACTED" "http://127.0.0.1:8870" 2ms
REDACTED - - [02/Jan/2025:04:20:39 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/1.1" 404 5755 "-" "Synapse/1.90.0" 2715 "REDACTED" "http://127.0.0.1:8870" 2ms
REDACTED - - [02/Jan/2025:04:20:40 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/1.1" 404 5755 "-" "Synapse/1.90.0" 2716 "REDACTED" "http://127.0.0.1:8870" 2ms
REDACTED - - [02/Jan/2025:04:20:40 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/1.1" 404 5755 "-" "Synapse/1.90.0" 2717 "REDACTED" "http://127.0.0.1:8870" 2ms
REDACTED - - [02/Jan/2025:04:20:40 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/1.1" 404 5755 "-" "Synapse/1.90.0" 2718 "REDACTED" "http://127.0.0.1:8870" 2ms
REDACTED - - [02/Jan/2025:04:20:41 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/1.1" 404 5755 "-" "Synapse/1.90.0" 2719 "REDACTED" "http://127.0.0.1:8870" 2ms
REDACTED - - [02/Jan/2025:04:20:41 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/1.1" 404 5755 "-" "Synapse/1.90.0" 2720 "REDACTED" "http://127.0.0.1:8870" 2ms
REDACTED - - [02/Jan/2025:04:20:42 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/1.1" 404 5755 "-" "Synapse/1.90.0" 2721 "REDACTED" "http://127.0.0.1:8870" 2ms

`Conduwuit/0.5.0`

Triggers one request, but will will retry every 5s-10m.

REDACTED - - [02/Jan/2025:04:09:33 +0000] "GET /.well-known/matrix/server HTTP/2.0" 200 40 "https://REDACTED/.well-known/matrix/server" "Conduwuit/0.5.0 (REDACTED)" 2656 "REDACTED" "http://127.0.0.1:9021" 1ms
REDACTED - - [02/Jan/2025:04:09:33 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/2.0" 404 5755 "-" "Conduwuit/0.5.0 (REDACTED)" 2657 REDACTED" "http://127.0.0.1:8870" 15ms
REDACTED - - [02/Jan/2025:04:09:46 +0000] "GET /.well-known/matrix/server HTTP/2.0" 200 40 "https://REDACTED/.well-known/matrix/server" "Conduwuit/0.5.0 (REDACTED)" 2658 "REDACTED" "http://127.0.0.1:9021" 1ms
REDACTED - - [02/Jan/2025:04:09:46 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/2.0" 404 5755 "-" "Conduwuit/0.5.0 (REDACTED)" 2659 REDACTED" "http://127.0.0.1:8870" 2ms
REDACTED - - [02/Jan/2025:04:19:04 +0000] "GET /.well-known/matrix/server HTTP/2.0" 200 40 "https://REDACTED/.well-known/matrix/server" "Conduwuit/0.5.0 (REDACTED)" 2690 "REDACTED" "http://127.0.0.1:9021" 1ms
REDACTED - - [02/Jan/2025:04:19:04 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/2.0" 404 5755 "-" "Conduwuit/0.5.0 (REDACTED)" 2691 REDACTED" "http://127.0.0.1:8870" 17ms
REDACTED - - [02/Jan/2025:04:19:20 +0000] "GET /.well-known/matrix/server HTTP/2.0" 200 40 "https://REDACTED/.well-known/matrix/server" "Conduwuit/0.5.0 (REDACTED)" 2693 "REDACTED" "http://127.0.0.1:9021" 1ms
REDACTED - - [02/Jan/2025:04:19:20 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/2.0" 404 5755 "-" "Conduwuit/0.5.0 (REDACTED)" 2694 REDACTED" "http://127.0.0.1:8870" 2ms
REDACTED - - [02/Jan/2025:04:24:46 +0000] "GET /.well-known/matrix/server HTTP/2.0" 200 40 "https://REDACTED/.well-known/matrix/server" "Conduwuit/0.5.0 (REDACTED)" 2729 "REDACTED" "http://127.0.0.1:9021" 2ms
REDACTED - - [02/Jan/2025:04:24:46 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/2.0" 404 5755 "-" "Conduwuit/0.5.0 (REDACTED)" 2730 REDACTED" "http://127.0.0.1:8870" 16ms
REDACTED - - [02/Jan/2025:04:25:17 +0000] "GET /.well-known/matrix/server HTTP/2.0" 200 40 "https://REDACTED/.well-known/matrix/server" "Conduwuit/0.5.0 (REDACTED)" 2731 "REDACTED" "http://127.0.0.1:9021" 1ms
REDACTED - - [02/Jan/2025:04:25:18 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/2.0" 404 5755 "-" "Conduwuit/0.5.0 (REDACTED)" 2732 REDACTED" "http://127.0.0.1:8870" 2ms
REDACTED - - [02/Jan/2025:04:34:38 +0000] "GET /.well-known/matrix/server HTTP/2.0" 200 40 "https://REDACTED/.well-known/matrix/server" "Conduwuit/0.5.0 (REDACTED)" 2781 "REDACTED" "http://127.0.0.1:9021" 1ms
REDACTED - - [02/Jan/2025:04:34:38 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/2.0" 404 5755 "-" "Conduwuit/0.5.0 (REDACTED)" 2782 REDACTED" "http://127.0.0.1:8870" 16ms
REDACTED - - [02/Jan/2025:04:34:42 +0000] "GET /.well-known/matrix/server HTTP/2.0" 200 40 "https://REDACTED/.well-known/matrix/server" "Conduwuit/0.5.0 (REDACTED)" 2783 "REDACTED" "http://127.0.0.1:9021" 1ms
REDACTED - - [02/Jan/2025:04:34:42 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/2.0" 404 5755 "-" "Conduwuit/0.5.0 (REDACTED)" 2784 REDACTED" "http://127.0.0.1:8870" 2ms
REDACTED - - [02/Jan/2025:04:41:10 +0000] "GET /.well-known/matrix/server HTTP/2.0" 200 40 "https://REDACTED/.well-known/matrix/server" "Conduwuit/0.5.0 (REDACTED)" 2807 "REDACTED" "http://127.0.0.1:9021" 1ms
REDACTED - - [02/Jan/2025:04:41:10 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/2.0" 404 5755 "-" "Conduwuit/0.5.0 (REDACTED)" 2808 REDACTED" "http://127.0.0.1:8870" 25ms
REDACTED - - [02/Jan/2025:04:41:12 +0000] "GET /.well-known/matrix/server HTTP/2.0" 200 40 "https://REDACTED/.well-known/matrix/server" "Conduwuit/0.5.0 (REDACTED)" 2809 "REDACTED" "http://127.0.0.1:9021" 1ms
REDACTED - - [02/Jan/2025:04:41:12 +0000] "PUT /_matrix/federation/v1/send/REDACTED HTTP/2.0" 404 5755 "-" "Conduwuit/0.5.0 (REDACTED)" 2810 REDACTED" "http://127.0.0.1:8870" 2ms

This is what I get on my end, not sure what "scope" this would have if I participated in more channels, servers or user messages. It also leaves me with the task of excluding 404s (or fake well known) everywhere or blocking the hosts in my firewall, which I did in the linked crowdsec issue, ending up with 200k dropped packets and 500+ ips over a few days.

Jan 02 '25 05:01 adrianrudnik

Thanks @adrianrudnik with those info I have managed to add a WAF rule on cloudflare for those specific locations (/.well-known/matrix, /_matrix) for all countries.

Jan 06 '25 00:01 CustomIcon