jeffgeerling-com
jeffgeerling-com copied to clipboard
geerlingguy
Document various DDoS attacks
So, shortly after my post Hosting this website on a farm - or anywhere went live, the site was pummeled by an average of 5,000,000 POST requests per hour (in addition to a ton of other traffic, I don't even have the full metrics because I turned some of that off once the server load was hitting 25-30).
There was no way the poor Turing Pi 2 cluster could handle that load—and not even my 2GB DigitalOcean VPS could.
So I had to turn to Cloudflare. I went 13 years without needing a hosted CDN/cache layer in front of my site, and went through dozens of HN front page sessions without an issue.
But when someone points a DDoS cannon at the site (I could confirm at least 1,000 unique IPs were sending dozens of requests per second), requiring Cloudflare is inevitable.
A few things I need to clean up after all this insanity:
- [ ] My automation needs to incorporate the new DigitalOcean firewall rules I added:
- [ ] HTTP 80/443: CloudFlare IP ranges, Solr, and Home IP
- [ ] TCP 4949: Munin monitoring
- [ ] SSH 22: Home IP
- [ ] ICMP: Any
- [x] I need to better configure/document/automate the new Cloudflare configuration (I kinda YOLO'ed everything to get it back online with a few page rules)
- [x] I need to document the entire process of getting back online (I have a number of screenshots which will be helpful in that regard)
- [x] I would like to get real ip addresses in my logs again... have to add some config for that
- [x] I have strict SSL cert mode set up in CloudFlare—I need to make sure I can still get valid certs through certbot behind CloudFlare—or use their cert maybe
- [x] Some people have mentioned getting duplicate RSS items (e.g. with NetNewsWire) after the Cloudflare switchover. I can't replicate it with Feedly, nor with
curl https://www.jeffgeerling.com/blog.xml, so I'm not sure what's happening for these users. (see https://github.com/geerlingguy/jeffgeerling-com/issues/145) - [x] Finish automating the edit domain restrictions in Nginx's config
- [x] Set up automated Cloudflare purging when posting a new blog post. (see https://github.com/geerlingguy/jeffgeerling-com/issues/146)
This took quite some time on my 4 GB access log file, but:
$ awk '{print $1}' access.log | sort | uniq -c | sort -nr
11326588 164.90.211.4
408590 162.158.62.94
386731 162.158.62.96
376055 108.162.219.25
372830 162.158.62.148
276436 127.0.0.1
275066 104.236.203.61
49038 108.162.219.67
40257 139.99.99.165
31197 51.79.144.52
30987 162.158.62.32
30786 173.245.52.202
28335 173.245.52.218
27393 217.16.188.90
25984 128.199.108.29
25235 143.208.200.26
23878 51.159.5.133
22145 108.162.219.129
21551 51.195.188.28
20793 108.162.219.63
20655 133.167.65.45
19872 173.245.52.214
19759 173.245.52.200
18515 133.167.121.133
18133 51.159.3.223
18061 160.16.82.58
17873 108.162.219.177
17848 54.36.176.76
17295 196.192.179.38
17277 159.203.181.211
16608 108.162.219.139
15876 89.40.13.33
15630 134.119.206.106
14842 194.233.69.38
14375 130.185.119.20
14055 35.152.75.76
13848 206.189.151.3
13450 185.199.119.73
13375 47.243.135.104
13344 45.154.255.147
13121 176.9.63.62
13066 92.42.109.186
12490 122.155.165.191
12481 131.255.239.38
12358 160.251.97.210
12283 66.94.120.161
12232 62.205.169.74
12071 103.6.52.35
12009 185.58.19.241
11841 167.86.81.208
11832 139.59.16.95
11799 45.129.56.200
11758 154.126.20.254
11650 200.25.48.72
11464 165.22.216.241
11279 182.253.93.4
11263 185.220.100.254
11254 110.232.67.42
11172 201.20.100.142
11066 209.9.37.60
11042 45.33.26.53
11032 8.215.27.71
11009 204.199.67.174
10835 185.220.100.241
10666 217.196.20.150
10505 92.51.95.194
10456 165.22.26.191
10445 177.129.253.133
10316 104.236.78.102
10038 138.201.120.214
10000 93.104.215.115
I only went to the first bunch that went to 10,000. Would be interesting to do a little IP research and see what all these IPs are doing. The first one looks to be a German DigitalOcean IP address.
I used the 'report abuse' feature on DigitalOcean to report 164.90.211.4, we'll see what comes of that.
The pattern here suggests DDoS as a service.
Such services leverage compromised hosts from their botnet networks to DDoS victims and each infected host is limited to around 1-5Mbps to avoid detection by their owners/users. So that's 1-5Mbps per host times a few million infected hosts.
So "report abuse" for in these cases, will probably not solve anything. DDoS as a service is growing exponentially every year.
I had a similar issue with my wife's wordpress some weeks ago. I first for i in $(cat logs | cut ... ); do iptables -I INPUT -s ... -j DROP, but since the patterns were really obvious in the logs (POST requests instead of GET, see below), I ended up with the following fail2ban custom configuration:
$ cat /etc/fail2ban/filter.d/wp-ddos.conf
[Definition]
failregex = ^<HOST> \- \S+ \[\] \"(POST //xmlrpc.php HTTP/1.1|POST /pattern2/ HTTP/1.1|POST /wp-content/ HTTP/1.1)\".+$
$ cat /etc/fail2ban/jail.d/wp-ddos.conf
[wp-ddos]
enabled = true
port = http,https
filter = wp-ddos
logpath = /var/log/nginx/access.log
maxretry = 2
findtime = 60
bantime = 604800
backend = pyinotify
So "report abuse" for in these cases, will probably not solve anything. DDoS as a service is growing exponentially every year.
True, but that one IP was orders of magnitude beyond the rest. Hopefully it will at least prompt DigitalOcean to clean up that server.
A few more quick stats since things are calming down (at least now that Cloudflare's handling the brunt of it:
And looking at my logs from the time before Cloudflare started working... I'm seeing 22,062,150 requests were made in that time frame (about 1-2 hours). On an average day (not just 1-2 hours), my website gets around 1,000,000 HTTP requests.
Can you post your log for the attack? I don't think we need all of it, but I would like to see if there is some sort of fail2ban and/or rate limiting rules that can be done. I mostly just need to see the IP address, time, and request method.
I would also look at Cactus Comments. It is a decentralized comment system based off of Matrix. That will allow you to have comments that don't break through the cache That should prevent some DDOS attacks on your site in the future as well. I don't know how easy it is to integrate into Drupal, and I haven't found a plugin for it, but you know more about how Drupal works then I do.
@minecraftchest1 - Eventually if I find a good comment system that integrates with spam prevention services that I can integrate with a static site and don't feel like it will turn out to be another Discord, I'd love to go that route and convert from Drupal to Hugo or some other static generator.
Well someone is at it again. Approximately 5 minutes ago I got an outage alert, and saw the exact same patterns. Hundreds of thousands of requests like:
150.129.148.88 - - [11/Feb/2022:23:09:04 -0600] "GET https://www.jeffgeerling.com/ HTTP/1.3" 200 11514 "https://www.jeffgeerling.com/" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3599.0 Safari/537.36" "-"
...
186.250.160.234 - - [11/Feb/2022:23:09:04 -0600] "POST / HTTP/1.2" 499 0 "https://www.jeffgeerling.com/" "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; yie11; rv:11.0) like Gecko" "-"
...
At least 150 unique IP addresses and here's a list of the top ones, sampling just a second or two of accesses:
$ awk '{print $1}' access.log | sort | uniq -c | sort -nr
64 185.220.100.244
57 103.120.175.243
56 144.91.85.172
53 78.37.24.148
44 164.70.119.206
40 158.140.170.183
32 45.177.108.162
32 139.228.65.97
31 185.220.101.37
31 103.135.220.91
27 180.246.196.111
22 142.11.199.242
21 201.150.117.164
21 127.0.0.1
19 94.16.15.100
15 104.236.203.61
15 103.175.80.3
14 87.255.13.217
12 88.132.34.230
12 66.94.97.238
11 123.115.115.213
11 119.148.32.25
11 111.68.31.156
11 103.101.17.170
10 165.16.27.13
Cloudflare IPs are popping into the list, since I switched it on again. Wish there were a simple plugin or something for Nginx to just pop on Cloudflare mode and only allow access from those IPs (without me having to pull down the list from Cloudflare itself).
For now, since I'm not on the site and I'm okay with basically disabling comments overnight, I've re-enabled my Cloudflare firewall rule to block all POST requests. The attack is not sophisticated... I might just add a rule in Nginx to block POST requests to anything that's not a valid comment, and see if that holds up (since I don't have the ability to get that deep on Cloudflare).
Seems like the exact same attack that plays out over on Nixcraft: https://www.cyberciti.biz/faq/nginx-block-post-requests-urls-for-spammer-ip-address-cidr/
Here's a fancy bandwidth graph on my server:
This might be of some help. https://stackoverflow.com/questions/47830088/force-cache-on-files-with-nginx I would also see if you can rate limit POST requests. I would also force the use of a cache using the above link. That will make the comments outdated, but will lower the load on your 4G connection.
@minecraftchest1 - Eventually if I find a good comment system that integrates with spam prevention services that I can integrate with a static site and don't feel like it will turn out to be another Discord, I'd love to go that route and convert from Drupal to Hugo or some other static generator.
I don't understand your comment about another Discord.
Since you are using GitHub, you can also look at https://utteranc.es/ as well. Just some options that may help lower attack surfaces.
@minecraftchest1 - I meant Disqus :D — basically, anything that I don't self-host will eventually have an incentive to jam ads and tracking into their product (even if they're a paid service, most likely).
@minecraftchest1 - I meant Disqus :D — basically, anything that I don't self-host will eventually have an incentive to jam ads and tracking into their product (even if they're a paid service, most likely).
That makes more sense. I created #143 to discuss this more as it is off topic for this thread.
Added a note that curl https://www.jeffgeerling.com/blog.xml returns the correct feed, but some users (notably, for NetNewsWire) are seeing duplicates of every article I post now. Not sure what's going on, but I also asked out loud on Twitter: https://twitter.com/geerlingguy/status/1498438448520253443
Edit: Someone else also noted this in newsboat, so it's not an isolated incident:
Opened new issue for the duplicate RSS feed items: https://github.com/geerlingguy/jeffgeerling-com/issues/145
Another attack after I posted the video How I survived a DDoS attack.
Nginx request log:
Set Cloudflare to 'under attack' mode for now since I have other things to do this morning :P
root@www:/var/log/nginx# awk '{print $1}' access.log | sort | uniq -c | sort -nr
119742 162.158.62.94
108478 162.158.62.96
103952 108.162.219.25
99629 162.158.62.148
24292 127.0.0.1
24100 104.236.203.61
20404 108.162.219.67
11709 173.245.52.202
11559 162.158.62.32
10885 173.245.52.218
10820 108.162.219.63
9829 108.162.219.129
8950 173.245.52.214
8742 162.158.63.61
7416 173.245.52.200
7055 108.162.219.139
6326 108.162.219.177
5168 162.158.62.178
2029 108.162.219.21
1321 162.158.63.59
1269 162.158.63.51
1225 108.162.219.153
1153 108.162.219.107
1089 162.158.62.56
1079 162.158.62.140
1039 173.245.52.206
1033 162.158.62.40
1014 173.245.52.208
Obviously most of the requests are coming through CF—I need to modify my command (will do later) to grab the real IP address.
This time the script is hitting /user/login, which I set to bypass (will have to switch that off, lol):
108.162.219.25 - - [16/Mar/2022:11:28:36 -0500] "GET /user/login HTTP/1.1" 200 2901 "https://www.jeffgeerling.com/user/login" "Mozilla/5.0 (Windows NT 6.1; Trident/7.0; Touch; rv:11.0) like Gecko" "66.187.4.202"
127.0.0.1 - - [16/Mar/2022:11:28:36 -0500] "GET /user/login HTTP/1.0" 200 8736 "https://www.jeffgeerling.com/user/login" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.7.1; rv:8.3) Gecko/20100101 Firefox/8.3.6" "162.158.62.148"
162.158.62.148 - - [16/Mar/2022:11:28:36 -0500] "GET /user/login HTTP/1.1" 200 2901 "https://www.jeffgeerling.com/user/login" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.7.1; rv:8.3) Gecko/20100101 Firefox/8.3.6" "176.214.34.58"
173.245.52.214 - - [16/Mar/2022:11:28:36 -0500] "GET /user/login HTTP/1.1" 200 2901 "https://www.jeffgeerling.com/user/login" "Mozilla/5.0 (Windows; U; Windows NT 5.3) AppleWebKit/533.1.0 (KHTML, like Gecko) Chrome/14.0.880.0 Safari/533.1.0" "2604:2dc0:200:1654::"
162.158.62.96 - - [16/Mar/2022:11:28:36 -0500] "GET /user/login HTTP/1.1" 200 2901 "https://www.jeffgeerling.com/user/login" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.5.9; rv:9.3) Gecko/20100101 Firefox/9.3.9" "201.149.103.113"
Decided to just block the /user/login path entirely for now. Nobody but me needs access to it ;)
One of the top IPs seems to belong to an elementary school... https://ipinfo.io/165.139.238.20
Probably a hacked teacher's computer.
Hundreds of thousands of requests blocked now:
Up to 1.3 million now.
Edit: 1.8 million after another minute. lol
Edit 2: 2.2 million after another minute
