website icon indicating copy to clipboard operation
website copied to clipboard
verified publisher icon caddyserver

Submitting plugin caused Caddy website to start spamming my server

Open mediocregopher opened this issue 10 months ago • 14 comments

Hello, sorry if this isn't the correct place for this bug report, I wasn't able to find a support email for the caddy website itself.

Today I signed up at https://caddyserver.com/account and submitted my plugin suite, dev.mediocregopher.com/mediocre-caddy-plugins.git. I use this set of plugins in my personal Caddy deployment successfully, so I'm quite sure that this import path works correctly. However upon submitting it I noticed the following things:

  • Bursts of requests like the following happening every 10 seconds or so: 
    {"level":"info","ts":1741534109.0850415,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"X.X.X.X","remote_port":"35464","client_ip":"X.X.X.X","proto":"HTTP/2.0","method":"GET","host":"dev.mediocregopher.com","uri":"/mediocre-caddy-plugins/info/refs?service=git-upload-pack","headers":{"Pragma":["no-cache"],"User-Agent":["git/2.25.1"],"Accept":["*/*"],"Accept-Encoding":["deflate, gzip, br"],"Accept-Language":["C, *;q=0.9"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"dev.mediocregopher.com"}},"bytes_read":0,"user_id":"","duration":0.002172244,"size":366,"status":200,"resp_headers":{"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"Cache-Control":["no-cache, max-age=0, must-revalidate"],"Content-Type":["application/x-git-upload-pack-advertisement"],"Expires":["Fri, 01 Jan 1980 00:00:00 GMT"],"Pragma":["no-cache"]}}
  • Various parts of the Caddy website seem to have stopped working, like the JSON Config Structure page and the Download page
  • Even after temporarily blocking all requests to https://dev.mediocregopher.com/mediocre-caddy-plugins/ (all return 500 for the moment) and removing the plugin from my account the bursts of git requests have continued. Some pages, like the Download page, started working again, but others have not.

I'm wondering if something about my plugin package caused some kind of bug which the site wasn't able to gracefully recover from?

mediocregopher avatar Mar 09 '25 15:03 mediocregopher

I'm confident it isn't Caddy server doing it because we don't do checks for updates. You censored the remote_ip, which is the IP address of the client calling your server, so we can't validate who'se doing it. There were reports of the Go module proxy making excessive calls, reported in golang/go#44577 and here.

mohammed90 avatar Mar 09 '25 16:03 mohammed90

The IP is 159.65.99.124, which WHOIS shows is owned by DigitialOcean, on which caddyserver.com is also hosted.

mediocregopher avatar Mar 09 '25 17:03 mediocregopher

The IP is 159.65.99.124, which WHOIS shows is owned by DigitialOcean, on which caddyserver.com is also hosted.

Hmm, yeah, that's our server. We'll investigate.

mohammed90 avatar Mar 09 '25 17:03 mohammed90

We do pull the code to do static analysis on it so we can display the docs. Our database is showing signs of stress currently. May be related.

mholt avatar Mar 09 '25 20:03 mholt

Hi all, has there been any movement on this? I've blocked that IP completely on my host, but I'm still seeing essentially constant connection requests from it, multiple times a second:

09:12:09.448073 enp4s0 In  IP 159.65.99.124.38738 > 192.168.1.9.80: Flags [S], seq 1864525182, win 64240, options [mss 1452,sackOK,TS val 1925710243 ecr 0,nop,wscale 7], length 0
09:12:10.439264 enp4s0 In  IP 159.65.99.124.46382 > 192.168.1.9.443: Flags [S], seq 1652502710, win 64240, options [mss 1452,sackOK,TS val 1925711235 ecr 0,nop,wscale 7], length 0
09:12:11.471811 enp4s0 In  IP 159.65.99.124.38738 > 192.168.1.9.80: Flags [S], seq 1864525182, win 64240, options [mss 1452,sackOK,TS val 1925712259 ecr 0,nop,wscale 7], length 0
09:12:11.471902 enp4s0 In  IP 159.65.99.124.41272 > 192.168.1.9.80: Flags [S], seq 1975603783, win 64240, options [mss 1452,sackOK,TS val 1925712263 ecr 0,nop,wscale 7], length 0
09:12:12.506092 enp4s0 In  IP 159.65.99.124.38740 > 192.168.1.9.80: Flags [S], seq 2023587641, win 64240, options [mss 1452,sackOK,TS val 1925713300 ecr 0,nop,wscale 7], length 0
09:12:12.514241 enp4s0 In  IP 159.65.99.124.46472 > 192.168.1.9.443: Flags [S], seq 302600101, win 64240, options [mss 1452,sackOK,TS val 1925713308 ecr 0,nop,wscale 7], length 0
09:12:13.256455 enp4s0 In  IP 159.65.99.124.59244 > 192.168.1.9.80: Flags [S], seq 3001610714, win 64240, options [mss 1452,sackOK,TS val 1925714051 ecr 0,nop,wscale 7], length 0
09:12:13.511387 enp4s0 In  IP 159.65.99.124.38740 > 192.168.1.9.80: Flags [S], seq 2023587641, win 64240, options [mss 1452,sackOK,TS val 1925714307 ecr 0,nop,wscale 7], length 0
09:12:13.543612 enp4s0 In  IP 159.65.99.124.46472 > 192.168.1.9.443: Flags [S], seq 302600101, win 64240, options [mss 1452,sackOK,TS val 1925714339 ecr 0,nop,wscale 7], length 0
09:12:14.565183 enp4s0 In  IP 159.65.99.124.38744 > 192.168.1.9.80: Flags [S], seq 3518398753, win 64240, options [mss 1452,sackOK,TS val 1925715360 ecr 0,nop,wscale 7], length 0
09:12:14.565218 enp4s0 In  IP 159.65.99.124.46486 > 192.168.1.9.443: Flags [S], seq 517585417, win 64240, options [mss 1452,sackOK,TS val 1925715360 ecr 0,nop,wscale 7], length 0
09:12:15.527757 enp4s0 In  IP 159.65.99.124.38740 > 192.168.1.9.80: Flags [S], seq 2023587641, win 64240, options [mss 1452,sackOK,TS val 1925716323 ecr 0,nop,wscale 7], length 0
09:12:15.559831 enp4s0 In  IP 159.65.99.124.38738 > 192.168.1.9.80: Flags [S], seq 1864525182, win 64240, options [mss 1452,sackOK,TS val 1925716355 ecr 0,nop,wscale 7], length 0
09:12:15.560189 enp4s0 In  IP 159.65.99.124.46472 > 192.168.1.9.443: Flags [S], seq 302600101, win 64240, options [mss 1452,sackOK,TS val 1925716355 ecr 0,nop,wscale 7], length 0
09:12:15.592104 enp4s0 In  IP 159.65.99.124.46486 > 192.168.1.9.443: Flags [S], seq 517585417, win 64240, options [mss 1452,sackOK,TS val 1925716387 ecr 0,nop,wscale 7], length 0
09:12:15.592104 enp4s0 In  IP 159.65.99.124.38744 > 192.168.1.9.80: Flags [S], seq 3518398753, win 64240, options [mss 1452,sackOK,TS val 1925716387 ecr 0,nop,wscale 7], length 0
09:12:17.353410 enp4s0 In  IP 159.65.99.124.35842 > 192.168.1.9.443: Flags [S], seq 686943747, win 64240, options [mss 1452,sackOK,TS val 1925718147 ecr 0,nop,wscale 7], length 0
09:12:17.609289 enp4s0 In  IP 159.65.99.124.46486 > 192.168.1.9.443: Flags [S], seq 517585417, win 64240, options [mss 1452,sackOK,TS val 1925718403 ecr 0,nop,wscale 7], length 0
09:12:17.609290 enp4s0 In  IP 159.65.99.124.38744 > 192.168.1.9.80: Flags [S], seq 3518398753, win 64240, options [mss 1452,sackOK,TS val 1925718403 ecr 0,nop,wscale 7], length 0
09:12:19.399736 enp4s0 In  IP 159.65.99.124.41258 > 192.168.1.9.80: Flags [S], seq 3821167491, win 64240, options [mss 1452,sackOK,TS val 1925720195 ecr 0,nop,wscale 7], length 0
09:12:19.655823 enp4s0 In  IP 159.65.99.124.38740 > 192.168.1.9.80: Flags [S], seq 2023587641, win 64240, options [mss 1452,sackOK,TS val 1925720451 ecr 0,nop,wscale 7], length 0
09:12:19.655823 enp4s0 In  IP 159.65.99.124.46472 > 192.168.1.9.443: Flags [S], seq 302600101, win 64240, options [mss 1452,sackOK,TS val 1925720451 ecr 0,nop,wscale 7], length 0
09:12:21.703721 enp4s0 In  IP 159.65.99.124.46486 > 192.168.1.9.443: Flags [S], seq 517585417, win 64240, options [mss 1452,sackOK,TS val 1925722499 ecr 0,nop,wscale 7], length 0
09:12:21.703722 enp4s0 In  IP 159.65.99.124.38744 > 192.168.1.9.80: Flags [S], seq 3518398753, win 64240, options [mss 1452,sackOK,TS val 1925722499 ecr 0,nop,wscale 7], length 0
09:12:22.728220 enp4s0 In  IP 159.65.99.124.54704 > 192.168.1.9.80: Flags [S], seq 1471341389, win 64240, options [mss 1452,sackOK,TS val 1925723523 ecr 0,nop,wscale 7], length 0
09:12:22.728220 enp4s0 In  IP 159.65.99.124.54708 > 192.168.1.9.80: Flags [S], seq 15033119, win 64240, options [mss 1452,sackOK,TS val 1925723523 ecr 0,nop,wscale 7], length 0
09:12:22.750118 enp4s0 In  IP 159.65.99.124.42782 > 192.168.1.9.80: Flags [S], seq 2607590442, win 64240, options [mss 1452,sackOK,TS val 1925723544 ecr 0,nop,wscale 7], length 0
09:12:23.751916 enp4s0 In  IP 159.65.99.124.38738 > 192.168.1.9.80: Flags [S], seq 1864525182, win 64240, options [mss 1452,sackOK,TS val 1925724547 ecr 0,nop,wscale 7], length 0
09:12:23.751916 enp4s0 In  IP 159.65.99.124.42782 > 192.168.1.9.80: Flags [S], seq 2607590442, win 64240, options [mss 1452,sackOK,TS val 1925724547 ecr 0,nop,wscale 7], length 0
09:12:25.769417 enp4s0 In  IP 159.65.99.124.42782 > 192.168.1.9.80: Flags [S], seq 2607590442, win 64240, options [mss 1452,sackOK,TS val 1925726563 ecr 0,nop,wscale 7], length 0
09:12:26.824121 enp4s0 In  IP 159.65.99.124.58438 > 192.168.1.9.443: Flags [S], seq 188396582, win 64240, options [mss 1452,sackOK,TS val 1925727619 ecr 0,nop,wscale 7], length 0
09:12:27.592444 enp4s0 In  IP 159.65.99.124.41272 > 192.168.1.9.80: Flags [S], seq 1975603783, win 64240, options [mss 1452,sackOK,TS val 1925728387 ecr 0,nop,wscale 7], length 0
09:12:27.848529 enp4s0 In  IP 159.65.99.124.38740 > 192.168.1.9.80: Flags [S], seq 2023587641, win 64240, options [mss 1452,sackOK,TS val 1925728643 ecr 0,nop,wscale 7], length 0
09:12:27.848530 enp4s0 In  IP 159.65.99.124.46472 > 192.168.1.9.443: Flags [S], seq 302600101, win 64240, options [mss 1452,sackOK,TS val 1925728643 ecr 0,nop,wscale 7], length 0
09:12:28.871139 enp4s0 In  IP 159.65.99.124.58448 > 192.168.1.9.443: Flags [S], seq 3276516735, win 64240, options [mss 1452,sackOK,TS val 1925729667 ecr 0,nop,wscale 7], length 0
09:12:28.871140 enp4s0 In  IP 159.65.99.124.58458 > 192.168.1.9.443: Flags [S], seq 611276895, win 64240, options [mss 1452,sackOK,TS val 1925729667 ecr 0,nop,wscale 7], length 0
09:12:29.896743 enp4s0 In  IP 159.65.99.124.38744 > 192.168.1.9.80: Flags [S], seq 3518398753, win 64240, options [mss 1452,sackOK,TS val 1925730691 ecr 0,nop,wscale 7], length 0
09:12:29.900145 enp4s0 In  IP 159.65.99.124.46486 > 192.168.1.9.443: Flags [S], seq 517585417, win 64240, options [mss 1452,sackOK,TS val 1925730695 ecr 0,nop,wscale 7], length 0
09:12:29.900146 enp4s0 In  IP 159.65.99.124.42782 > 192.168.1.9.80: Flags [S], seq 2607590442, win 64240, options [mss 1452,sackOK,TS val 1925730695 ecr 0,nop,wscale 7], length 0
09:12:30.920008 enp4s0 In  IP 159.65.99.124.54710 > 192.168.1.9.80: Flags [S], seq 66221504, win 64240, options [mss 1452,sackOK,TS val 1925731715 ecr 0,nop,wscale 7], length 0
09:12:32.968132 enp4s0 In  IP 159.65.99.124.34870 > 192.168.1.9.443: Flags [S], seq 574342701, win 64240, options [mss 1452,sackOK,TS val 1925733763 ecr 0,nop,wscale 7], length 0
09:12:32.968177 enp4s0 In  IP 159.65.99.124.34874 > 192.168.1.9.443: Flags [S], seq 155706570, win 64240, options [mss 1452,sackOK,TS val 1925733763 ecr 0,nop,wscale 7], length 0

It's reached the point where it's affecting my home internet quality (where my server is hosted), I think my ISP is likely throttling me. For now I'm going to blackhole my dev.mediocregopher.com domain, which should alleviate the problem for me temporarily, but obviously that isn't a long term solution.

mediocregopher avatar Mar 12 '25 08:03 mediocregopher

Yes, but work is ongoing. @Mohammed90 seems to have resolved the core issue (related to Prometheus) and now I'm upgrading some things on the server.

mholt avatar Mar 12 '25 11:03 mholt

@mediocregopher Is the issue still occurring?

mholt avatar Mar 12 '25 13:03 mholt

@mholt Sorry I've been out all day, but yes it is still occurring:

22:01:12.315262 enp4s0 In  IP 159.65.99.124.45128 > 192.168.1.9.443: Flags [S], seq 3670496171, win 64240, options [mss 1452,sackOK,TS val 3496717476 ecr 0,nop,wscale 7], length 0
22:01:12.315292 enp4s0 In  IP 159.65.99.124.48808 > 192.168.1.9.80: Flags [S], seq 3434575847, win 64240, options [mss 1452,sackOK,TS val 3496717476 ecr 0,nop,wscale 7], length 0
22:01:12.315317 enp4s0 In  IP 159.65.99.124.45114 > 192.168.1.9.443: Flags [S], seq 1659537461, win 64240, options [mss 1452,sackOK,TS val 3496717476 ecr 0,nop,wscale 7], length 0
22:01:12.315341 enp4s0 In  IP 159.65.99.124.48812 > 192.168.1.9.80: Flags [S], seq 3264301898, win 64240, options [mss 1452,sackOK,TS val 3496717476 ecr 0,nop,wscale 7], length 0
22:01:13.906524 enp4s0 In  IP 159.65.99.124.42204 > 192.168.1.9.80: Flags [S], seq 3401149499, win 64240, options [mss 1452,sackOK,TS val 3496719268 ecr 0,nop,wscale 7], length 0
22:01:14.362541 enp4s0 In  IP 159.65.99.124.38376 > 192.168.1.9.80: Flags [S], seq 4144155440, win 64240, options [mss 1452,sackOK,TS val 3496719524 ecr 0,nop,wscale 7], length 0
22:01:16.458689 enp4s0 In  IP 159.65.99.124.45128 > 192.168.1.9.443: Flags [S], seq 3670496171, win 64240, options [mss 1452,sackOK,TS val 3496721572 ecr 0,nop,wscale 7], length 0
22:01:16.463414 enp4s0 In  IP 159.65.99.124.45114 > 192.168.1.9.443: Flags [S], seq 1659537461, win 64240, options [mss 1452,sackOK,TS val 3496721572 ecr 0,nop,wscale 7], length 0
22:01:17.482138 enp4s0 In  IP 159.65.99.124.53628 > 192.168.1.9.443: Flags [S], seq 3807329901, win 64240, options [mss 1452,sackOK,TS val 3496722596 ecr 0,nop,wscale 7], length 0
22:01:17.483966 enp4s0 In  IP 159.65.99.124.53614 > 192.168.1.9.443: Flags [S], seq 17692555, win 64240, options [mss 1452,sackOK,TS val 3496722596 ecr 0,nop,wscale 7], length 0
22:01:17.484691 enp4s0 In  IP 159.65.99.124.53606 > 192.168.1.9.443: Flags [S], seq 1095248707, win 64240, options [mss 1452,sackOK,TS val 3496722596 ecr 0,nop,wscale 7], length 0
22:01:18.267039 enp4s0 In  IP 159.65.99.124.44062 > 192.168.1.9.443: Flags [S], seq 3651560875, win 64240, options [mss 1452,sackOK,TS val 3496723364 ecr 0,nop,wscale 7], length 0
22:01:21.879052 enp4s0 In  IP 159.65.99.124.55996 > 192.168.1.9.443: Flags [S], seq 415721824, win 64240, options [mss 1452,sackOK,TS val 3496726731 ecr 0,nop,wscale 7], length 0

mediocregopher avatar Mar 12 '25 21:03 mediocregopher

Did it let up today at all? We had several hours where it was working better.

mholt avatar Mar 12 '25 21:03 mholt

Sorry I'm not sure, for the most part I left dev.mediocregopher.com being blackhole'd so I wasn't receiving traffic for it =/ I can leave it not-blackhole'd for tonight, with tcpdump running, and I can post the results tomorrow if that helps.

mediocregopher avatar Mar 12 '25 21:03 mediocregopher

Nah, don't worry about it -- the issue has returned already, so I am still trying to figure out what is causing it.

mholt avatar Mar 12 '25 22:03 mholt

@mediocregopher Does your plugin have any circular references to its own types, perchance? i.e. a pointer of a type that contains a pointer to the same type. (Might be any number of levels of indirection)

Just trying to figure out what could possibly cause a loop...

Weird thing is the code that actually calls "git" isn't even in our code base, it's in "go get", which we do invoke, but only on claiming a package or package rescan.

mholt avatar Mar 13 '25 17:03 mholt

@mholt nothing obvious regarding loops comes to mind. The only thing that does come to mind is that I have a module which exposes a git repo over https using the gitkit project: https://dev.mediocregopher.com/mediocre-caddy-plugins/tree/http/handlers/git_remote_repo.go

I can't really imagine how this would result in your server trying to clone the repo though.... it should only have to do with local repos.

In any case, do you think you could take my repo out of your db (or whatever is causing it to continue to be processed)? You could put a clone of it into github or whatever and try to repro off of that, but I would really like to clear out my firewall tables and make my ISP happy.

mediocregopher avatar Mar 14 '25 09:03 mediocregopher

@mediocregopher I just checked, and it's not even in our database.

Unfortunately I can't connect to the DB since there's already too many connections. I'm not sure where they're coming from, at first we suspected a leak from Prometheus, but supposedly that was patched and it's still happening. So now we're looking into other possibilities.

I'll shut down the website backend for now, since it's not serving any useful content at the moment anyway. (This will break our download page temporarily though.)

(We may run it sometimes as we troubleshoot.)

mholt avatar Mar 14 '25 15:03 mholt