Cgit is currently unavailable

[nikromen]

• copr-dist-git crashed, reboot ended up with the "press ctrl+D" prompt, mounted the data volume to the STG instance, run fsck manually, remounted back, did chmod -x on the cgit binary.

Originally posted by @FrostyX in #15

[praiskup]

FTR, here is the recent botnet attack @ Savannah: https://lists.gnu.org/archive/html/savannah-users/2025-01/msg00000.html

[praiskup]

170212
[root@copr-dist-git httpd][PROD]# grep 14/Jan ssl_access_log-20241208  | grep cgit | wc -l
10998
[root@copr-dist-git httpd][PROD]# grep 15/Jan ssl_access_log-20241208  | grep cgit | wc -l
11387
[root@copr-dist-git httpd][PROD]# grep 16/Jan ssl_access_log-20241208  | grep cgit | wc -l
9834
[root@copr-dist-git httpd][PROD]# grep 17/Jan ssl_access_log-20250118  | grep cgit | wc -l
161555
[root@copr-dist-git httpd][PROD]# grep 18/Jan ssl_access_log-20250118  | grep cgit | wc -l
170212

2025-01-17 and 20215-01-18 shows significantly higher throughput.

This is also interesting:

$ grep 17/Jan ssl_access_log-20250118  | grep cgit | cut -d' ' -f1 | sort | uniq -c
... first ~100 users have at most 14 accesses ...
      6 77.111.111.111
      9 193.92.200.156
      9 2a00:23c6:6e8d:fb01:509a:af42:c511:3834
     11 80.109.108.129
     14 2003:cf:d729:dc99:7119:b8a1:80da:ec3b
    262 47.79.99.1
    265 47.79.121.89
    267 47.79.123.136
    269 47.82.0.13
    269 47.82.0.98
... the rest ~500 users have ~250-350 accessess ... all in 47.79.x.x - 47.82.x.x...

IOW, similar issue to Savannah

[praiskup]

See also #3595.

[praiskup]

see also: https://fosstodon.org/@[email protected]/113868168298182149

[mkurz]

Any ETA when cgit will be up again?

[FrostyX]

@mkurz I think somebody will start working on this next week, but I have no idea how hard it will be to solve this. We think we were DDoSed by AI scrapers like many other projects (please see the mastodon link above). And the last time I checked, it wasn't so clear what to do about it.

Does the disabled cgit break some of your workflows? Maybe there will be a workaround that you can use until it is fixed.

[mkurz]

Any news?

[FrostyX]

We discussed it last week and there was an idea about restoring the previous status quo and simply enabling the cgit instance again. The problem is, we expect to get DDoSed again. And once it happens, it will disable the whole machine, causing build queue to not get processed and so on. And if it happens during the night, the outage could be for many hours. So this is probably not worth the gamble.

Other idea was to limit the cgit resources through cgroups. Once DDoSed, it will probably shutdown just the cgit instance, leaving the rest of the system unimpaired

Lastly, we discussed using some kind of AI tarpit, but we don't want to maintain it. Since this is as global issue, we hoped maybe Fedora Infra team could come up with a solution for the whole Fedora ecosystem.

[step21]

Is there in the meantime a way to access repos? I am wondering because afaict, the patched mesa-krunkit source for example does not seem to be available anywhere else, or not that I can find.

[FrostyX]

@step21 no worries, there is. Use the exact same repo URL that you have but change /cgit/ to /git/. You will be able to clone it.

[step21]

ok, thx.

[praiskup]

Nice post: https://www.paritybit.ca/blog/choosing-a-self-hosted-git-service/

[nikromen]

a lot of services/websites added some "I am not a robot" checks like https://github.com/google/recaptcha

not that it is bulletproof, but it may decrease the load to the cgit

[praiskup]

There's a (non-Fedora) RPM https://anubis.techaro.lol/docs/admin/native-install that we could experiment with.

[nikromen]

arch linux wiki faces the same problem as us and they protect their servers with https://github.com/TecharoHQ/anubis

it seems to be lightweigth and (relatively) easy to deploy solution... more reading on how anubis works and how to configure it: https://anubis.techaro.lol/docs/

however it seems to be designed to block almost every scraper, even the small ones like internet archive bot, but I don't think this is concern to us since we want this to be available for our users as frontend for git, nothing more

but afaik if we decide to go this way we can go with cloudfare instead https://www.cloudflare.com/application-services/products/turnstile/ - I think it's open-source and free, if not then anubis is the way probably

EDIT: ah I see that I am late to the party since @praiskup saw yesterday the same bot blocking software as me today :D

[FrostyX]

We should be careful if we go the Cloudflare route. I've heard too many stories that somebody's traffic spiked due to some DDoS and their bill skyrocketed. That's doesn't mean we shouldn't use it, I am just saying we should be careful about the limits.

[ferdnyc]

@FrostyX

You sure? Tons of people decamped from Netlify to Cloudflare when Netlify presented a "free" user with a $100,000 bill for DDoS-driven overages. (Didn't help that their CEO then parachuted in, and completely failed to improve their PR image with a ham-handed notsplanation.) But I haven't seen reports of Cloudflare doing the same thing.

[japotta08]

} https://documentation.ubuntu.com/server/ {

[ferdnyc]

Eagerly (that's a lie) awaiting (that too) the explanation for how Ubuntu Server is magically immune to DDoS attacks.

[nikromen]

since a few people are watching progress of this: we are trying to push anubis to fedora... some progress was already made: https://bugzilla.redhat.com/show_bug.cgi?id=2360490 and packaging/review of a few dependencies is needed for this, which I am trying to help with to sped up this process

[purpleidea]

On this page where we see:

The cgit interface is temporarily unavailable We had to temporarily disable the cgit interface for Copr DistGit because it was causing performance issues. Copr builders clone the repositories from a different URL (and you can do too) so building packages is unaffected. Ticket: #3591

Could we specify the URL to git clone that data? For example, I tried to access:

https://copr-dist-git.fedorainfracloud.org/cgit/sunwire/dkms-r8168/dkms-r8168.git

And I have no idea what the proper git link should be.

Thanks

[FrostyX]

@purpleidea I was a bit paranoid and intentionally not mentioned the URL on the website, in case somebody wanted to DDoS that one as well. Which would be much worse, because that would affect every Copr build.

However, you can just change cgit in that URL to git and you will be able to clone it.

[purpleidea]

@FrostyX That's very helpful, thanks! I certainly wouldn't mind seeing something like:

However, you can just change cgit in that URL to git and you will be able to clone it.

On the page for future stumblers like me!

Cheers

[praiskup]

Team meeting time: Jirka discussed with Davide; we had a problem in his package anubis.rpm being reviewed https://bugzilla.redhat.com/show_bug.cgi?id=2360489 - and we seem to be blocked by one more build-time dependency (css-nano package is missing). Jirka seems to have a work-around now. Or please correct me if I don't understand well.

[nikromen]

yes, what happenned is that:

• packaging of anubis is tracked here https://bugzilla.redhat.com/show_bug.cgi?id=2360489
• we needed to package some dependencies to fedora for anubis
  • so I reviewed https://bugzilla.redhat.com/show_bug.cgi?id=2360490
  • and packaged these: https://bugzilla.redhat.com/show_bug.cgi?id=2371327 https://bugzilla.redhat.com/show_bug.cgi?id=2371335 https://bugzilla.redhat.com/show_bug.cgi?id=2387591
  • last dependency missing for anubis is https://bugzilla.redhat.com/show_bug.cgi?id=2387591 - already in review process
  • also a bug was found in postcss-cli package that block building https://bugzilla.redhat.com/show_bug.cgi?id=2360489#c5 but known workaround/fix is already known
• I created tmp repo in copr that fixes all the bug that I encoutered with (and are not shipped in fedora yet) and it seems to build, so hopefully once these will land to fedora, anubis can also finish its review process https://copr.fedorainfracloud.org/coprs/nikromen/anubis-test/build/9398644/

[nikromen]

The POC is running on dev: https://copr-dist-git-dev.fedorainfracloud.org/cgit/ - it still isn't in fedora, so the final configuration will probably slightly differ, but it's at least doing something right now.

the config currently used: https://pagure.io/fedora-infra/ansible/pull-request/2801

[nikromen]

anubis landed last week into fedora 41, however with broken binary. this update https://bodhi.fedoraproject.org/updates/FEDORA-2025-65df8e12ae somewhat made it functional, however it still need some patches. I'll continue with those in different issues, since the anubis is deployed in dist-git and cgit is working once again.

If you encounter any issues with it, please create an issue or ping me here.