docker-gitlab
docker-gitlab copied to clipboard
sameersbn
Gitlab CPU Utilization is HIGH due to XMRig tool
Hi Team, We are facing issue regarding GITLAB, (version 13.11.4) CPU is very high due to XMRig tool, we are killing the process but it starts automatically even after killing the process, how can we find the source and fix this issue? what are the possibilities that it can create the issue?
Mine was also mined. The version is: gitlab/gitlab-ee:13.5.3-ee.0, I temporarily solved the problem, but I don't know why.
Mine was also mined. The version is: gitlab/gitlab-ee:13.5.3-ee.0, I temporarily solved the problem, but I don't know why.
How do you solved the problem?
how do you find the source of causing and what is the solution that u have followed?
我的也被开采了。版本是：gitlab/gitlab-ee:13.5.3-ee.0，我暂时解决了这个问题，但不知道为什么。
你是如何解决这个问题的？
My gitlab is run using docker-compose, and the data is mounted locally. The way I handle it is very simple and rude. I first made a backup and then deleted the attacked container, and then re-run gitlab with docker-compose.
我的也被开采了。版本是：gitlab/gitlab-ee:13.5.3-ee.0，我暂时解决了这个问题，但不知道为什么。
你是如何解决这个问题的？
My gitlab is run using docker-compose, and the data is mounted locally. The way I handle it is very simple and rude. I first made a backup and then deleted the attacked container, and then re-run gitlab with docker-compose.
After re-run does the issue came back?
On April 14, 2021, GitLab published a security release to address CVE-2021-22205, a critical remote code execution vulnerability in the service’s web interface. At the time, GitLab described the issue as an authenticated vulnerability that was the result of passing user-provided images to the service’s embedded version of ExifTool. A remote attacker could execute arbitrary commands as the git user due to ExifTool’s mishandling of DjVu files, an issue that was later assigned CVE-2021-22204.
https://attackerkb.com/topics/D41jRUXCiJ/cve-2021-22205/rapid7-analysis?referrer=blog
我的也被开采了。版本是：gitlab/gitlab-ee:13.5.3-ee.0，我暂时解决了这个问题，但不知道为什么。
你是如何解决这个问题的？
My gitlab is run using docker-compose, and the data is mounted locally. The way I handle it is very simple and rude. I first made a backup and then deleted the attacked container, and then re-run gitlab with docker-compose.
After re-run does the issue came back?
Yes, he came back one day after another. I am going to upgrade.
then what is the permanent solution? did you find any solution?
I upgraded the version to 13.8.8
Hello Everyone. Just wanted to add some details that may be helpful as it helped in my case.
1- All GitLab versions except 13.8.8, 13.9.6, 13.10.3 are vulnerable with remote code execution using a default user called git. 2- only GitLab container has this bug hence underlying system has no role to play here. 3- Looks like it has been globally accepted and published on CVE on 1st November. 4- Probably these look C**nese miners and use CPU mining for Monero cryptocurrency using XMRig tool. 5- Here are more details for the interested to readout for this vulnerability. (https://www.rapid7.com/blog/post/2021/11/01/gitlab-unauthenticated-remote-code-execution-cve-2021-22205-exploited-in-the-wild/)
Precautions and notes
- First please perform the following steps in a non-production environment and observe the issue.
- Please take the complete backup of your entire GitLab data before performing any steps.
- The steps mentioned here are not the official notes from Gitlab or any other organization.
- We hold no responsibility for any data loss caused during the steps.
How to find impacted container ?
Just run the docker stats command and it will tell you which container is eating CPU. It should be your GitLab container.
How to see malware in action?
- Login to your impacted container.
- Check out
ls -la /tmpdirectory and see what are the executable directories or files you found, which you are not running intentionally or unaware. - Check whats are the cron jobs scheduled using
crontab -l -u git - Since miners are using various methods, sites to download malware scripts, dynamic process names e.g xmrig, agetty, xyz so it may be difficult to prevent this attack using Firewall, ACL, IP blocking etc.
What's the solution then?
- Just use docker-compose down and up: You are safe for a few hours as miners will come back and reinsert their code because your GitLab is still vulnerable.
- Just restart the running container: You are safe only till the cron job scheduled time comes into the round/place.
Are you looking for a permanent solution?
Perform GitLab upgrade from your current version to patched version: 13.8.8, 13.9.6, 13.10.3.
Can't perform an upgrade due to other dependencies or issues? Ok then try out the following solution.
1- Login to the infected GitLab container.
2- Use the top command and kill the unwanted process eating most of the CPU.
3- sudo rm -rf /tmp/* (It will cleanup already downloaded existing virus code files).
4- sudo chmod -R 600 /tmp (It will prevent them to re-execute code from this directory), assuming you are not using /tmp directory for any other intentional purpose, else you may need to figure out a way to manage it for your scenario.
5- touch /etc/cron.allow (It will prevent miners from running any cron job). But if you are running any intentional cron job like backup, restore, use different user from git, and mention that username name in /etc/cron.allow file.
6- Repeat step 2 as cron job may again run the process till the time you perform steps 4 and 5.
crontab -l -u git print nothing and nothing in /var/spool/cron/crontabs/.
I'm using gitlab 13.7.4-ee, and it was injected. There are '/tmp/spread*' files. It gained execute permission over and over again no matter how much time I kill the corresponding process. And I decided to upgrade the container. Thanks for open this issue.
