containerd
containerd copied to clipboard
containerd
1.5.10 causes memory leak in mysql container; regression from 1.4.13
Description
I updated containerd.io on Fedora from 1.4.13 to 1.5.10. After this, all mysql docker containers across multiple projects began using 20+GB of memory where before they used <300MB. I restarted the computer and confirmed the problem persisted. I then rolled back to 1.4.13 and the problem was gone.
Steps to reproduce the issue
- Run a docker container with linux/amd64 arch with
ENV MYSQL_VERSION=5.7.36-0ubuntu0.18.04.1 - Start mysqld inside the container
- Observe massive memory usage in 1.5.10
Describe the results you received and expected
No regression
What version of containerd are you using?
1.5.10
Any other relevant information
runc version 1.0.3 commit: v1.0.3-0-gf46b6ba spec: 1.0.2-dev go: go1.16.15 libseccomp: 2.5.3
Linux localhost.localdomain 5.16.7-200.fc35.x86_64 #1 SMP PREEMPT Sun Feb 6 19:53:54 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Show configuration if it is related to CRI plugin.
No response
I have the same issue
runc version 1.0.3
commit: v1.0.3-0-gf46b6ba
spec: 1.0.2-dev
go: go1.16.15
libseccomp: 2.5.3
I implemented a temp solution using ulimit like this comment. It's worked for me. https://github.com/docker-library/mysql/issues/579#issuecomment-1074069882
The difference may be due to this PR: https://github.com/containerd/containerd/pull/4475
I have the same issue
runc version 1.0.3 commit: v1.0.3-0-gf46b6ba spec: 1.0.2-dev go: go1.16.15 libseccomp: 2.5.3I implemented a temp solution using ulimit like this comment. It's worked for me. docker-library/mysql#579 (comment)
Yes, can confirm that this is a valid workaround. It seems very likely that the increase to infinity nofile size is the problem on some systems.
That's .... interesting. If that's causing an issue, that feels like a bug in systemd 🤔 (as infinity should be equivalent to the old value)
I will do a repro test later this week with 1.5 and 1.6, after modifying the service file, to see if the problem is isolated to just that line
Thanks in advance for that!
Slightly orthogonal; I recall I once opened a ticket (more a "reminder"); to look into other options in this area; that ticket was related to the systemd unit for docker-ce, but the same applies to the one for containerd (which was originally based on what we had in docker); https://github.com/docker/for-linux/issues/73. If anyone arriving here is interested in doing some research into those options (also taking into account if they're supported on all "usual" distros), that would be welcome.
I installed 1.5 and confirmed problem still persisted. I then modified /lib/systemd/system/containerd.service to read LimitNOFILE=1048576. I then restarted the service and the daemon: sudo systemctl restart service containerd sudo systemctl daemon-reload. Problem still persisted. I then restarted the service once more, because I thought the daemon might have cached the service (so in total I restarted service, daemon, service) and the problem was gone. Docker stable repos don't have 1.6 available so I didn't test that.
I think you should do daemon-reload first and then restart containerd. And it only makes sure that it works for new containers, not for existing ones.
I applied the same fix as @C-Higgins and it also got rid of the issue on my machine. (containerd version 1.5.11 )
I installed 1.5 and confirmed problem still persisted. I then modified
/lib/systemd/system/containerd.serviceto readLimitNOFILE=1048576. I then restarted the service and the daemon:sudo systemctl restart service containerdsudo systemctl daemon-reload. Problem still persisted. I then restarted the service once more, because I thought the daemon might have cached the service (so in total I restarted service, daemon, service) and the problem was gone. Docker stable repos don't have 1.6 available so I didn't test that.
Not work for me
https://github.com/docker-library/mysql/issues/840
апр 11 08:40:09 tk-kv.localhost kernel: eth0: renamed from vethe040851
апр 11 08:40:09 tk-kv.localhost kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethf1fa9b5: link becomes ready
апр 11 08:40:09 tk-kv.localhost kernel: docker0: port 1(vethf1fa9b5) entered blocking state
апр 11 08:40:09 tk-kv.localhost kernel: docker0: port 1(vethf1fa9b5) entered forwarding state
апр 11 08:40:09 tk-kv.localhost NetworkManager[702]:
Hi, its work for me. MYSQL
docker system prune -a nano /lib/systemd/system/containerd.service LimitNOFILE=1048576 systemctl daemon-reload systemctl restart containerd
and not work
gitlab-runner -> docker -> containerd 1.5.11
yarn install апр 12 08:40:52 debug3.local gitlab-runner[1068]: Checking for jobs... nothing runner=rccAVBWo апр 12 08:40:54 debug3.local kernel: webpack invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 апр 12 08:40:54 debug3.local kernel: CPU: 15 PID: 3018610 Comm: webpack Not tainted 5.16.18-200.fc35.x86_64 #1 апр 12 08:40:54 debug3.local kernel: Hardware name: Micro-Star International Co., Ltd. MS-7A38/B450M PRO-VDH MAX (MS-7A38), BIOS B.70 06/10/2020
debug3.local kernel: Memory cgroup out of memory: Killed process 3017931 (webpack) total-vm:38547452kB, anon-rss:3423448kB, file-rss:2896kB, shmem-rss:0kB, UID:1000 pgtables:85972kB oom_score_adj:0
Seems there is a bug in Systemd, if you need more than 65535 files, need to change the LimitNOFILE manually to a bigger number, such as 1048576
https://github.com/systemd/systemd/issues/6559
Maybe we need to revert this PR 4475
It works fine on Ubuntu 18.04.6 but it fails on Fedora 35 running same containerd and runc version
go1.16.15 containerd: 1.5.11 runc: 1.0.3
These commands crashes on Fedora 35 because of container use too much memory: docker run -it --rm mysql:5.7.36 docker run -it --rm mysql:5.5.62
This command works fine on Fedora 35: docker run -it --rm mysql:8.0.29
It works fine on Fedora 35 running mysql:5.x after setting LimitNOFILE to 1048576 in containerd.service as @kangclzjc said
Hello. I'm experiencing the same issue.
Launching mysql:8.* works great, but mysql:5.7.* causes immediate 100% memory consumption (htop), and results in the following in /var/log/messages:
2022-06-06T17:23:24.094275-04:00 laptop kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/system.slice/docker-xxx.scope,task=mysqld,pid=38421,uid=0
2022-06-06T17:23:24.094288-04:00 laptop kernel: Out of memory: Killed process 38421 (mysqld) total-vm:16829404kB, anon-rss:12304300kB, file-rss:108kB, shmem-rss:0kB, UID:0 pgtables:28428kB oom_score_adj:0
2022-06-06T17:23:24.094313-04:00 laptop systemd[1]: docker-xxx.scope: A process of this unit has been killed by the OOM killer.
2022-06-06T17:23:24.856029-04:00 laptop systemd[1]: docker-xxx.scope: Deactivated successfully.
Versions:
Docker version 20.10.16, build aa7e414
containerd containerd.io 1.6.4 212e8b6fa2f44b9c21b2798135fc6fb7c53efc16
runc version 1.1.1
commit: v1.1.1-0-g52de29d
spec: 1.0.2-dev
go: go1.17.9
libseccomp: 2.5.3
Linux laptop1138 5.17.12-200.fc35.x86_64 #1 SMP PREEMPT Mon May 30 16:58:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Fedora release 35 (Thirty Five)
Limits:
cat /proc/$(pidof dockerd)/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size unlimited unlimited bytes
Max resident set unlimited unlimited bytes
Max processes unlimited unlimited processes
Max open files 1073741816 1073741816 files
Max locked memory 8388608 8388608 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 62780 62780 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
ulimit -a
real-time non-blocking time (microseconds, -R) unlimited
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 62780
max locked memory (kbytes, -l) 8192
max memory size (kbytes, -m) unlimited
open files (-n) 64000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 62780
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
systemctl show containerd | grep LimitNOFILE
LimitNOFILE=infinity
LimitNOFILESoft=infinity
Creating the following custom limit for containerd does not resolve the issue. /etc/systemd/system/containerd.service.d/custom.conf
[Service]
LimitNOFILE=1048576
Note: Prior systemctl show containerd | grep LimitNOFILE would report 'unlimited'.
systemctl show containerd | grep LimitNOFILE
LimitNOFILE=1048576
LimitNOFILESoft=1048576
This change did not resolve the issue as suggested here: https://github.com/containerd/containerd/issues/3201
Flushing all the docker components and reinstalling resolved the issue. (new containerd version released 22h ago)
sudo dnf remove docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-selinux \
docker-engine-selinux \
docker-engine
sudo dnf -y install dnf-plugins-core
sudo dnf config-manager \
--add-repo \
https://download.docker.com/linux/fedora/docker-ce.repo
sudo dnf install docker-ce docker-ce-cli containerd.io docker-compose-plugin
# REF: https://docs.docker.com/engine/install/fedora/
containerd --version
containerd containerd.io 1.6.6 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
Thanks!
What worked for me on Fedora 36: sudo vim /lib/systemd/system/containerd.service ->
#LimitNOFILE=infinity
LimitNOFILE=1048576
Then:
systemctl daemon-reload
sudo systemctl restart containerd
docker ps -a # (find id and then docker rm id)
run again: docker run --name=mysql1 -d mysql/mysql-server:5.7
After that it worked for me, where with the value set to unlimited, the container crashed in seconds. This change does not seem to be needed on Ubuntu and derivates, is there anything else messing this up?
Launching mysql:8.* works great, but mysql:5.7.* causes immediate 100% memory consumption (htop), and results in the following in /var/log/messages:
MySQL is having https://bugs.mysql.com/bug.php?id=96525.
Just to give this topic a little push here. I had the same problem. In our docker-compose file we need 3 different mysql servers and as they had no memory limits set they killed my whole system (fedora 37). A simple docker-compose up led to a system freeze and only a hard reset helped.
The change in /lib/systemd/system/containerd.service by setting the LimitNOFILE=1048576 helped!
Having the same problem running mysql 5.5 on a Kind cluster on Fedora 36. I haven't found a way to set the ulimit in kubernetes but I saw MariaDB fixed this bug on 5.5 so I'm just using MariaDB now and it's working fine.
I am also having the same issue on Fedora. Having to change the LimitNOFILE every time I run updates.
I'm having the same issue on Manjaro (Kernel 6.1.12-1), and changing LimitNOFILE=1048576 change made it work again.
So to summarize, this seemingly helped me on Fedora 37
sudo nano /lib/systemd/system/containerd.service
LimitNOFILE=INFINITY --> LimitNOFILE=1048576
docker stop <db>
docker rm <db>
sudo systemctl daemon-reload
sudo systemctl restart containerd
docker-compose up -d
@ThatCoffeeGuy thank you! This solved the same problem in my distro ( Manjaro linux )!
Just a heads-up that the LimitNOFILE=infinity setting in both docker.service and containerd.service files has finally been removed from their respective projects.
Possibly those changes will be part of new releases before the end of the year:
- v25 for Docker (with backports planned to
20.10) - ~~v1.7.8~~
v2.0for containerd?
After that happens, this issue can be considered resolved?
Just a heads-up that the
LimitNOFILE=infinitysetting in bothdocker.serviceandcontainerd.servicefiles has finally been removed from their respective projects.Possibly those changes will be part of new releases before the end of the year:
* v25 for Docker (_with backports planned to `20.10`_) * v1.7.8 for containerd?After that happens, this issue can be considered resolved?
I suppose so. FYI: I'm not very familiar with the Linux kernel or developing Linux apps.
If this is an issue of resource allocation, I suppose that would resolve it. If this is actually a memory leak or some bit of bad code, it might be worthwhile to dive into the two versions noted here and see if the code causing the issue can be identified.
I ran into issues trying to revert my containerd binaries, but I might try again.
If this is an issue of resource allocation, I suppose that would resolve it.
I haven't tried to reproduce the mysql one, but it's been a problem that affected many software where outside of containers the expected limits are considerably lower.
Default is 1024 soft limit. docker.service and containerd.service have overrided that to infinity which can be as high as over 1 billion. Usually that just stalls the software for a much longer duration (sometimes hours), but some software affected by this may also do some allocation, which as you can guess 1e3 vs 1e9 is quite the delta, a million times more. So if your software would normally allocate 1MB in this scenario and that allocation code was affected by this much large limit increase, it'd now use 1TB.
Normally it's just CPU from tasks like iterating through the limit range and closing potentially (highly unlikely) open file descriptors.
If this is actually a memory leak or some bit of bad code, it might be worthwhile to dive into the two versions noted here and see if the code causing the issue can be identified.
True. In some projects affected that I used, I helped track down the problem area and implement a workaround. However outside of container environments where the limits are sane and typically only raised when necessary, you're not likely to encounter this sort of problem. The bug is more to do with misconfiguration with LimitNOFILE=infinity being applied to the containers (and each process in those containers being run with a soft limit well above the expected 1024).
In the meantime, you can start the container with docker run --ulimit "nofile=1024:524288" extra option, or with systemd services set a drop-in override with systemctl edit containerd.service with the following:
[Service]
LimitNOFILE=1024:524288
- You may need to do the same for
docker.serviceuntil a release is out with that updated too. - After applying the edit for these files, restart Docker
systemctl restart docker. - You can verify these are the new limits applied with
docker run --rm -it alpine ash -c 'ulimit -Sn && ulimit -Hn', you should get1024and524288.
I wanted to add my own experience with this issue: apparently I can only reproduce it if swap is enabled. Starting the container with no swap (a.k.a running swapoff <device>) makes the container work just fine, and I can then reenable swap later
