Strange memory management (docker)

[mblazic]

qBittorrent version and Operating System

qBittorrent version: 4.3.0.1 Operating System: GNU/Linux CentOS 7.8.2003

If on linux, libtorrent-rasterbar and Qt version

libtorrent-rasterbar version: 1.2.10.0 Qt version: 5.9.5

What is the problem

I'm running qBittorrent client in a Docker container (version 19.03.13-3), at the moment I have 89 torrents loaded in the client. When a seed/leech process is started everything is normal, but as time passes RAM usage will reach memory limit for a qBittorrent container (4GB) and swap space will be continued to be used instead.

Client disk cache size is set to 3GB, which should be find for my torrent workload. The problem starts when the cache limit is reached (total buffer size is 2.99GB), then for some reason RAM usage is going up to 4GB (limit for a qBittorrent container) and then host swap file is being used to it's limit. Disk cache expiry interval is set to 600 seconds.

On a Windows machine , where the same version of qBittorrent client is being used, same as client disk cache size, RAM usage is fine balanced by qBittorrent . When a big size torrent is being seeded qBittorrent process will hit it's disk cache limit and wont go above it. When a torrent becomes idle, RAM will be released and client will use cca 15-20MB of memory.

What is the expected behavior

Maybe I'm doing something wrong, but I don't understand why qBittorrent client is using so much RAM inside a container, when a disk cache limit in my case is set to 3GB. On first, it looks like a memory leak. I had a same issue with version 4.2.5.

Steps to reproduce

No special steps, just run qBittorrent client inside a container to seed or download a torrent.

Extra info(if any)

Content save location is NFS (version 4.2) mount. No specific mounting flags are being used.

Settings:

I know this is not a place for Docker related issues, but maybe someone can give me some advice or a hint. Thank you in advance.

---

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[FranciscoPombal]

See https://github.com/qbittorrent/qBittorrent/issues/11783#issuecomment-718837272. There is no reason to use 3 GiB cache in your use-case. Can you reproduce this with a more reasonable cache amount, like 256 MiB?

[mblazic]

The idea behind so big disk cache size is slow mechanical disk and a lot of active torrents in the same time in the future. I have a fast connection, so I'm trying to pull out maximum out of it by offloading some of I/O operations from disk. 3 GiB of RAM as a disk cache size was looking as a sweet spot in my case, but after reading your comment, I'm probably thinking in a wrong way.

Thank you for a reply. Will try with 256 MiB of disk cache size and report back if I still experience the same issue.

[mblazic]

After 4 days of qBittorrent being started with 256 MiB of disk cache and light traffic, usage of RAM is at the moment at 500 MB. I haven't seen any memory usage drops when a client is idle for a few hours, compared to a Windows, where client RAM usage after a few more tests drops to 15-20MB. I have a feeling that RAM usage will go to it's maximum value/limit a time passes, but this time it will take a bit longer, because of lower disk cache size.

Maybe at some point, memory usage will stop increasing and start to flote around some value, but I can't confirm that at the moment. There is 4 GiB if memory available, so if it will help, I can let a client run for a few more days without any config changes or container restarts to see if memory usage will go up.

[FranciscoPombal]

Nice, please let us know further results of your testing.

If I understand correctly, you are claiming that on Windows, qBittorrent effectively frees unused cache memory after a period of inactivity, but the same doesn't happen on Linux (at least in docker). Is this correct?

[mblazic]

That's right. In qBittorrent client statistics, "Total buffer size" is at 255 MiB, and it's floating around that value. After monitoring it for a few days, I can confirm that it does not get lower when qBittorrent client is idle (no activity), sometimes it will drop to 240 MiB and of course it will get to 255 MiB again, when there is some torrent traffic activity. From the client perspective, it seams that disk cache value is being respected (from qBittorrent statistics data), but from the other side memory usage is rising.

Another part of this potential issue is that memory usage after 6 days without docker container restart is at 585 MiB. Maybe there are two corelated potential issues present, but I can't confirm that at this point.

[FranciscoPombal]

@mblazic

Sounds like there could be some kind of memory leak unrelated to the disk cache. If you can, you could try tracking it down with Valgrind's massif.

[mblazic]

I will try to track it down, and report back.

One think to mention, after two weeks of running docker container with qBittorrent client, memory usage is floating between 1 GiB and 2 GiB, but never under 1 GiB. I'm not sure if this value will go up with more torrents active in the same time. Also I'm not sure how memory releasing is working now, because for example at the moment all torrents are idle (no network activity) and memory usage is at 1.5 GiB. While monitoring memory usage today, I can confirm that it's not dropping.

[hnz101]

I really hope you find something. I am trying to switch to qBittorrent (4.3.01) on Debian (ZFS) since this week. But there is some strange behavior. First I also thought there is some kind of memory leak so I limited memory usage with systemd, but it has to be something else. Even with swappiness=1 and 16 Gb of memory the systems starts swapping.

I can't see any unusual high load on CPU, memory or I/O and yet qBitorrent stresses the system somehow on a level that I can't even watch a Movie lag free from an samba share on this machine.

[mblazic]

@hnz101

interesting, thanks for info. I'm using ZFS on CentOS 7.8.2003 (ESXi virtual machine) along with NFS to get access to the content for a qBittorrent client, which is running as a service in a docker container (ESXi virtual machine). In a process of testing qBittorrent client, version 4.2.5 I have used Hyper-V virtual machine and it's local ext4 storage (local image on a Windows machine). The result was the same as I have now, memory was not released after torrents went idle and memory usage went above disk cache limit value. From qBittorrent client statistics, "Total buffer size" value was not above disk cache limit value, so I think something else is the problem.

Maybe used memory is released at some point, but we don't notice that because of some sort of a memory leak that is bigger then a cache limit size itself.

If you have some time, you can run a test with Valgrind's massif too. I will try to do the same, plus a few more tests with qBittorrent client version 4.3.0.1 inside Hyper-V virtual machine.

[hnz101]

[...] If you have some time, you can run a test with Valgrind's massif too. [...]

I did today, but perhaps I need to add some parameter? The output wasn't really much after half a day running. I ran valgrind --tool=massif /usr/local/bin/qbittorrent-nox and got:

$ cat massif.out.32115 desc: (none) cmd: /usr/local/bin/qbittorrent-nox time_unit: i #----------- snapshot=0 #----------- time=0 mem_heap_B=0 mem_heap_extra_B=0 mem_stacks_B=0 heap_tree=empty

P.S. I'm running

Qt: 5.15.1 Libtorrent: 1.2.10.0 Boost: 1.74.0 OpenSSL: 1.1.1h zlib: 1.2.11

[Eraser3]

I run qBittorrent-nox 4.3.0.1 on a Proxmox LCX with Ubuntu 20.04.1 and I was also having problems with the memory. I ran out of memory (2GB) in about 15 minutes. I found this thread and set Disk Cache to 256MB and my problems seem to have been resolved. The previous settings was -1 by the way. That probably means 'unlimited'? My test is running for less than an hour so it might be soon to say but I can see in the host that memory is being released every now and then although the trend goes slightly upward. I will keep running with these settings for a couple of days to make sure it doesn't go crazy again and then I'll try to increase the Disk Cache slightly because I hope it will give me a better seeding performance.

[mblazic]

@hnz101 @Eraser3 Thank you on your comments regarding this issue, I will run and comment my results later this week, so we can continue forward with this discussion.

@Eraser3

The previous settings was -1 by the way. That probably means 'unlimited'?

Yes, that should be right.

[mblazic]

I have run two tests with Valgrind's massif tool, on a test virtual machine (Hyper-V) on GNU/Linux CentOS 8.2.2004 (kernel: 4.18.0-193.19.1.el8_2.x86_64). These tests were executed with qBittorrent v4.2.5 (nox) from EPEL repository.

First test, with disk cache set to 256MB: http://shorturl.at/klGZ5

Second test, with disk cache set to 3GB: http://shorturl.at/DOWZ9

I still need to setup a new virtual machine with GNU/Linux CentOS 7, on which I'm experiencing memory issue, and share Valgrind results.

[mblazic]

I ran valgrind --tool=massif /usr/local/bin/qbittorrent-nox and got:

$ cat massif.out.32115 desc: (none) cmd: /usr/local/bin/qbittorrent-nox time_unit: i #----------- snapshot=0 #----------- time=0 mem_heap_B=0 mem_heap_extra_B=0 mem_stacks_B=0 heap_tree=empty

@hnz101

Try to become root, and run: valgrind --tool=massif /usr/bin/qbittorrent-nox

since:

[root@lab-vm ~]# which qbittorrent-nox
/usr/bin/qbittorrent-nox

[Eraser3]

I let my client run for more than 5 days with Disk Cache set to 256MB. The client has uploaded over 800GB since that time. My memory has been stabilized at 1.1GB. I have no idea what the reference is between the used 1.1GB and the set Disk Cache of 256MB but I am glad that the client does not run out of memory any more. I will try to increase the Disk Cache slightly to see what the effects will be. The TOP command gives me the following output.

top - 09:37:56 up 7 days, 17:41,  1 user,  load average: 0.80, 0.96, 0.96
Tasks:  25 total,   1 running,  24 sleeping,   0 stopped,   0 zombie
%Cpu(s):  5.5 us,  0.0 sy,  0.0 ni, 94.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   2048.0 total,      4.2 free,   1156.0 used,    887.9 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.    892.0 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
 2239 qbittor+  20   0 1655124   1.0g   8772 S   6.0  51.5 400:11.16 qbittorrent-nox

I don't know if it's useful but I don't think I have many more information to share.

[mblazic]

My memory has been stabilized at 1.1GB. I have no idea what the reference is between the used 1.1GB and the set Disk Cache of 256MB but I am glad that the client does not run out of memory any more.

Yes, for now that's the main question, if we find the reference which we can follow or maybe custom tune it, we would be able to report a bug or change settings to respect a proper way for qBittorrent client configuration.

[FranciscoPombal]

@mblazic

Looks like those captures are a bit broken.

if we find the reference

The "normal" memory usage depends on many factors besides disk usage. Connected peers, number of torrents, search/RSS usage, etc. At the end of the day, if you suspect of a memory leak issue, you have to get your hands dirty with a profiler and/or debugger and look at the data structures to confirm/reject the hypothesis.

You could also try using this: https://github.com/google/orbit if you can't get massif to work. Seems nice, but I have no experience with it.

[tjjh89017]

I faced some similar issue with libtorrent It seems it's a bug in libtorrent version between 1.2.5 and 1.2.9

https://github.com/arvidn/libtorrent/issues/5332

[mblazic]

Thank you for info @tjjh89017. Please keep us informed with your findings.

[tjjh89017]

@mblazic It will take a long time to have something new from my side. EZIO is side project. and my job task is full until next year