unbound
unbound copied to clipboard
NLnetLabs
Unbound 1.13.1 .. 1.16.1 memory leak (complete RAM exhaustion) due to high volume of TCP FD's
Unbound 1.13.1 .. 1.16.1 memory leak (complete RAM exhaustion) due to high volume of TCP FD's
After upgrading public DoT from unbound 1.12.0-1~bpo10+1 to 1.13.1-1 (debian 11 default) we ran into severe RAM exhaustion by unbound process. Starting unbound with
outgoing-num-tcp: 8192 incoming-num-tcp: 16384 , the process starts up with a few hundred MB of RAM (RSS). Without any load or queries, the processes RSS starts to increase every 10 seconds almost exactly by 15168KB (the step is always between 15164 and 15172KB). This increase doesn't stop, until 99% of the hosts physical RAM is depleted - basically eating up 5.2GB of RAM every hour. On a 32GB RAM host, it took less than 6 hours to eat up all the resources. At 99% the RSS increase stops, process is still able to serve requests. As there aren't any queries being sent to unbound, the caches are empty - flushing every flusheable command with unbound-control doesn't make any difference.
unbound-control reload triggers oom-killer to clean up the mess:
Out of memory: Killed process 6869 (unbound) total-vm:40762004kB, anon-rss:32467148kB, file-rss:0kB, shmem-rss:0kB, UID:108 pgtables:79328kB oom_score_adj:0 Detailed oom-killer output is in the attach.
I built a debian deb package from latest unbound 1.16.1 release and it behaves exactly the same. Configure options are debian defaults from https://salsa.debian.org/dns-team/unbound
To reproduce Steps to reproduce the behavior: 1.Almost bare minimum configuration to trigger the issue - increase *-tcp-num and add a few threads:
server:
verbosity: 1
use-syslog: yes
interface: 0.0.0.0
interface: ::0
do-ip4: yes
do-ip6: yes
do-udp: yes
do-tcp: yes
outgoing-num-tcp: 8192
incoming-num-tcp: 16384
chroot: "/var/lib/unbound"
username: "unbound"
directory: "/var/lib/unbound"
access-control: 0.0.0.0/0 allow
access-control: 0::/0 allow
extended-statistics: yes
hide-identity: no
hide-version: yes
so-reuseport: yes
num-threads: 4
2.start unbound and observe process memory usage increase every 10 seconds. no queries/clients/requests needed. 3.wait for RAM depletion (5GB per hour) and execute 'unbound-control reload' to trigger oom-killer
Expected behavior Unbound process should start up with consuming required memory for TCP FDs and not to eat up all the available RAM resources afterwards. With 1.12 we could serve ~20K concurrent DoT connections for months (until we hit tcp sockets depletion bug) and constant 60% RAM usage level, but with 1.13+ unbound we ran out RAM within hours.
System:
- Unbound version: verified with 1.13.1 and 1.16.1
- OS: debian 11.2 with 5.10.0-9-amd64 kernel
-
unbound -Voutput:
Version 1.13.1
Configure line: --build=x86_64-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-option-checking --disable-silent-rules --libdir=${prefix}/lib/x86_64-linux-gnu --libexecdir=${prefix}/lib/x86_64-linux-gnu --disable-maintainer-mode --disable-dependency-tracking --disable-rpath --with-pidfile=/run/unbound.pid --with-rootkey-file=/var/lib/unbound/root.key --with-libevent --with-pythonmodule --enable-subnet --enable-dnstap --enable-systemd --with-chroot-dir= --with-dnstap-socket-path=/run/dnstap.sock --libdir=/usr/lib
Linked libs: libevent 2.1.12-stable (it uses epoll), OpenSSL 1.1.1k 25 Mar 2021
Linked modules: dns64 python subnetcache respip validator iterator
Version 1.16.1
Configure line: --build=x86_64-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-option-checking --disable-silent-rules --libdir=${prefix}/lib/x86_64-linux-gnu --runstatedir=/run --disable-maintainer-mode --disable-dependency-tracking --with-pythonmodule --with-pyunbound --enable-subnet --enable-dnstap --enable-systemd --with-libnghttp2 --with-chroot-dir= --with-dnstap-socket-path=/run/dnstap.sock --disable-rpath --with-pidfile=/run/unbound.pid --with-libevent --enable-tfo-client --with-rootkey-file=/usr/share/dns/root.key --enable-tfo-server
Linked libs: libevent 2.1.12-stable (it uses epoll), OpenSSL 1.1.1n 15 Mar 2022
Linked modules: dns64 python subnetcache respip validator iterator
TCP Fastopen feature available
I have noticed, that in some rare occurrences the memory RSS increasing stops at some level, but 9 service startups out of 10 are quaranteed highway towards oomkiller. Side note, that we use multiple IP addresses on the hosts loopback interface.
unbound_oomkilled.txt unbound_ram_graph
This configuration does not make memory increase for me. It starts, and then the memory stays exactly the same for several minutes.
Could you run unbound with verbosity: 4, so that it logs what it is doing?
Lets calculate how much memory that is supposed to use. Every thread holds tcp buffers for incoming and outgoing. The buffer size is msg-buffer-size in unbound.conf. That defaults to about 64k, with some more bytes on it. That is 8192 * 66k + 16384 * 66k times 4 for the threads, with 66k for a bit more overhead for other data, probably slightly too high. That is 1.6G x 4 threads, about 6.4 Gb of memory.
The process starts with only a couple of hundred Mb? That must be because of some sort of VM memory system. Is swapping on? And unbound has pushed a lot of its memory to swap? As it is using it, somehow, even though you say there are no queries, the VM pressure increases, and the RSS goes up. I do not know how that VM manages the RSS.
But it is also not 15G that you say it rises to. That seems a bit more, and I wonder how that could happen. It should not cause an issue on the 32G machine. I would like to see what unbound is doing, so the log output could be helpful. You upgraded a lot of versions in one go, so I am not sure if there is some code change specifically that is causing this. Supposedly, if there are no queries, unbound is doing nothing, and thus memory can not increase. This is what it does when I start it here.
My actual intention was to upgrade from 1.12 to 1.13.1 (default releases for debian 10 and 11). The 1.13.1 went RAM crazy with the same config we were running stable with unbound 1.12. So trying unbound 1.16.1 was to check, if there a change for better on latest unbound release, but no. Something changed with 1.13 that causes this RAM consumption.
I have two 32GB ram production hosts, which have in config:
outgoing-num-tcp: 8192
incoming-num-tcp: 8192
num-threads: 4
Using the formula You gave, I got max 4.1GB in total for TCP sockets, so the 30+GB shouldn't happen. Still both have run into complete RAM exhaustion and I don't use any swap device (Swap 0.0 total). The production configuration has TLS enabled on port 853.
MiB Mem : 32112.8 total, 726.4 free, 31122.1 used, 264.2 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 622.5 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15428 unbound 20 0 35.4g 30.2g 7348 S 0.0 96.4 24:53.16 unbound
MiB Mem : 32112.8 total, 245.2 free, 31758.0 used, 109.6 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 40.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14998 unbound 20 0 43.4g 30.8g 3244 S 0.0 98.3 0:12.23 unbound
On the devel host (8GB RAM vm) unbound process climbed during the night up to 6.4GB and this value matches the buffer calculation value You noted above.
MiB Mem : 7956.8 total, 209.3 free, 6579.8 used, 1167.8 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 1121.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
50466 unbound 20 0 10.3g 6.3g 7452 S 0.0 80.9 0:01.00 unbound
With outgoing-num-tcp: 8192 and incoming-num-tcp: 16384 the unbound starts up with 822MB of RSS and starts increasing by 15MB every 10 seconds (hopefully only until 6.4GB). Somehow on the production units, the threshold is above the sky. 10minutes after startup the unbound process is consuming ~1.8GB RSS and constantly increasing. It takes a minute after startup to begin increasing. In the attach is the verbosity 4 output for unbound (quite short log file): unbound_verbosity_4_log.txt
I even attached unbound process to strace (disabled threading by num-threads: 1). Strace log collects the startup chit-chat and the strace output stops, while the actual process proceeds to consume RSS memory.
The logs you added are similar to the logs I get when I start it. And then the memory usage does not change.
So unbound is not doing anything but the memory increases. I do not know what is happening, but it must involve something else on the machine.
The RSS starting at less than a couple Gb already means somehow it is not using the memory that unbound allocated.
In 1.13 the new feature is stream reuse, for upstream streams they are kept around to be used later. This also keeps around the SSL context. That also has buffers and uses space. Perhaps this is the problem somehow? There are also fixes for TCP stream out of order processing I think. Specifically also event handling.
If it is the upstream query reuse of older streams. That would use more memory. There is an option tcp-reuse-timeout that controls it. It defaults to 60 seconds. If this is the problem, then every time it opens a new TLS connection for another query, it makes a new connection and stores it in the reuse tree, where the 1.12 version would have used an existing buffer, from the RSS, and used that to connect. If you set the reuse time to 0 or something very small, then it would not cycle through the entire 16384 outgoing buffer space, which is maybe the issue? If it is not doing anything, with that empty log file, then it is also not doing upstream queries, so that does not change anything for that case.
Status update: I reverted one node back to Debian 10 (4.19.0-13-amd64), which was running before unbound 1.12.0 just fine and upgraded unbound to 1.13.1-1~bpo10+1. Result is OK! unbound is running fine, process started right at 2.7GB RSS and caches are building up on top of that 2.7GB. Heres a screenshot of resource usage for last 6 hours.
The major change is newer Debian 11 with 5.10.0-16-amd64. The TCP socket memory allocation is totally different as the unbound process starts up with less than 1GB of RSS and should grow up to the memory amount based on the formula above + the caches with overhead. So we expect the growth to "calm" down at the same level as with previous deb10, but unfortunately this doesn't happen and RAM exhaustion is inevitable.
Current test results:
- Debian 10 4.19.0-13-amd64 + unbound 1.12.0-1~bpo10+1 = OK
- Debian 10 4.19.0-13-amd64 + unbound 1.13.1-1~bpo10+1 = OK
- Debian 11 5.10.0-16-amd64 + 1.13.1-1 = NOK
- Debian 11 5.10.0-16-amd64 + 1.16.1 = NOK
I gathered pmap outputs for unbound 1.13.1 on debian 10 and debian 11. The output amounts are very similar and overall there are no significant differences, but the total processes RSS is on deb10 ~2.7GB and on deb11 ~30GB. pmap doesn't reveal, where's the excessive 27G. pmap_deb10_unbound_1.13.1.txt pmap_deb11_unbound_1.13.1.txt
Memory debugging isn't my cup of tea, but I'll do my best to solve this puzzle. All debugging tips are welcome.
We have a very similar issue with AdGuard DNS servers. Since we upgraded to Debian 11, unbound 16.1 started to leak.
Well... the leakage eventually stops, when the kernel has given to the unbound process "enough" inactive memory. So far I have tried on debian 11 different kernels:
- 5.10.0-16-amd64
- 5.18.0-0.bpo.1-amd64 from bullseye backports Various different TCP socket values, threads number, disabled/enabled TLS, different interface configurations schemes. I even started up a VM with 128GB RAM to find out - does it ever stop "leaking"? Well, it does stop - depending on the amount of sockets and threads. I don't know the exact formula, but the "theoretical max of the formula above" must be multiplied by some serious value.
outgoing-num-tcp: 8192
incoming-num-tcp: 16384
num-threads: 4
Stopped increasing 41.7GB ! That's a totally idle process without any actual cache data. Enabling/disabling TLS doesn't have any effect. There something very different with kernel 5.x that unbound can't manage properly. Are there any other distros with 5.x kernel to try with?
Here's a log of different configurations and the total memory consumption of unbound process when the process RSS stopped increasing (I use: 'vmstat -aw 10' to detect end of increasing). By switching the configuration from 10 interface: lines to
interface-automatic: yes
interface-automatic-ports: "53 853"
brought the memory usage down from 41.7GB to only 18GB. interface-automatic-ports is available since 1.16.0 and solves our tricky interface setup by just 2 lines.
Maybe this helps: unbound_interfaces_conf_changes.txt
Is this even unbound specific? Since unbound performs no activities, and then memory increases. The number of file descriptors opened seems to make the memory change. There is also do-ipv4 and do-ipv6 and do-udp and do-tcp options that you could experiment with to see the memory change. There may not be working DNS service while those options are toggled. There are also some socket options set on them, and some can be controlled like the rcv and snd buffer sizes, reuseport, by setting their options in the config file.
If it is just opening a socket that makes memory go strange, other software would have the same issue?
This is somehow very Debian 11 (Bullseye) specific. I installed a Fedora release 36 (Thirty Six) which has the same kernel version, copied the identical unbound config and it doesn't consume memory.
- Debian 11 : kernel 5.18.2-1~bpo11+1 and unbound 1.16.1
- Fedora 36 : kernel 5.18.13-200.fc36.x86_64 and unbound 1.16.0
On fedora it starts up with 1.5GB RSS and stays there rock solid (even dnsperf tcp tests don't matter):
MiB Mem : 7943.2 total, 5451.7 free, 1714.9 used, 776.7 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5993.0 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
798 unbound 20 0 18.5g 1.5g 8752 S 0.0 18.7 1:34.72 unbound
do-ipv6: no - doesn't improve the situation on debian 11. do-tcp: no - forbids tcp socket openings, process starts up only with UDP listener and doesn't consume any memory. The issue is strongly tcp sockets and debian 11 related.
When unbound consumed 41.7GB of RSS I calculated the memory consumption per TCP socket from RSS: RSS / threads / (outgoing sockets + incoming sockets) 43728408/4.0/(8192+16384) => 444.8 K well... it's 7 times more than theoretical max 66k.
I noticed a difference between debian and fedora tcp listening ports with netstat -lnp Both have 4 threads in config.
[root@fedora ~]# netstat -lnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:53 0.0.0.0:* LISTEN 798/unbound
tcp 0 0 0.0.0.0:853 0.0.0.0:* LISTEN 798/unbound
tcp6 0 0 :::53 :::* LISTEN 798/unbound
tcp6 0 0 :::853 :::* LISTEN 798/unbound
debian:~$ sudo netstat -lnp
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:53 0.0.0.0:* LISTEN 22508/unbound
tcp 0 0 0.0.0.0:53 0.0.0.0:* LISTEN 22508/unbound
tcp 0 0 0.0.0.0:53 0.0.0.0:* LISTEN 22508/unbound
tcp 0 0 0.0.0.0:53 0.0.0.0:* LISTEN 22508/unbound
tcp 0 0 0.0.0.0:853 0.0.0.0:* LISTEN 22508/unbound
tcp 0 0 0.0.0.0:853 0.0.0.0:* LISTEN 22508/unbound
tcp 0 0 0.0.0.0:853 0.0.0.0:* LISTEN 22508/unbound
tcp 0 0 0.0.0.0:853 0.0.0.0:* LISTEN 22508/unbound
tcp6 0 0 :::53 :::* LISTEN 22508/unbound
tcp6 0 0 :::53 :::* LISTEN 22508/unbound
tcp6 0 0 :::53 :::* LISTEN 22508/unbound
tcp6 0 0 :::53 :::* LISTEN 22508/unbound
tcp6 0 0 :::853 :::* LISTEN 22508/unbound
tcp6 0 0 :::853 :::* LISTEN 22508/unbound
tcp6 0 0 :::853 :::* LISTEN 22508/unbound
tcp6 0 0 :::853 :::* LISTEN 22508/unbound
The reason of increasing RES memory is stricktly related to Transparent Huge Pages (THP). I noticed, that every 10 seconds unbound processes AnonHugePages is increased by ~15MB. Output of sudo pmap -XX PID:
Tue 13 Sep 2022 08:55:52 AM UTC
2101: /usr/sbin/unbound -d -p -c /var/lib/unbound/etc/unbound/unbound.conf
Address Perm Offset Device Inode Size KernelPageSize MMUPageSize Rss Pss Shared_Clean Shared_Dirty Private_Clean Private_Dirty Referenced Anonymous LazyFree AnonHugePages ShmemPmdMapped FilePmdMapped Shared_Hugetlb Private_Hugetlb Swap SwapPss Locked THPeligible ProtectionKey VmFlags Mapping
555bc5570000 r--p 00000000 fd:00 402530 88 4 4 88 88 0 0 88 0 88 0 0 0 0 0 0 0 0 0 0 0 0 rd mr mw me sd unbound
555bc5586000 r-xp 00016000 fd:00 402530 924 4 4 724 724 0 0 724 0 724 0 0 0 0 0 0 0 0 0 0 0 0 rd ex mr mw me sd unbound
555bc566d000 r--p 000fd000 fd:00 402530 452 4 4 324 324 0 0 324 0 324 0 0 0 0 0 0 0 0 0 0 0 0 rd mr mw me sd unbound
555bc56df000 r--p 0016e000 fd:00 402530 8 4 4 8 8 0 0 0 8 8 8 0 0 0 0 0 0 0 0 0 0 0 rd mr mw me ac sd unbound
555bc56e1000 rw-p 00170000 fd:00 402530 44 4 4 44 44 0 0 0 44 44 44 0 0 0 0 0 0 0 0 0 0 0 rd wr mr mw me ac sd unbound
555bc56ec000 rw-p 00000000 00:00 0 4 4 4 4 4 0 0 0 4 4 4 0 0 0 0 0 0 0 0 0 0 0 rd wr mr mw me ac sd
555bc6ce7000 rw-p 00000000 00:00 0 8532104 4 4 1878384 1878384 0 0 0 1878384 1878384 1878384 0 1335296 0 0 0 0 0 0 0 1 0 rd wr mr mw me ac sd [heap]
7f327fb7e000 rw-p 00000000 00:00 0 1268 4 4 1268 1268 0 0 0 1268 1268 1268 0 0 0 0 0 0 0 0 0 0 0 rd wr mr mw me ac sd
======= ============== =========== ======= ======= ============ ============ ============= ============= ========== ========= ======== ============= ============== ============= ============== =============== ==== ======= ====== =========== =============
8552408 564 564 1888696 1884420 4856 0 2932 1880908 1888696 1880908 0 1335296 0 0 0 0 0 0 0 1 0 KB
+20 sec
2101: /usr/sbin/unbound -d -p -c /var/lib/unbound/etc/unbound/unbound.conf
Address Perm Offset Device Inode Size KernelPageSize MMUPageSize Rss Pss Shared_Clean Shared_Dirty Private_Clean Private_Dirty Referenced Anonymous LazyFree AnonHugePages ShmemPmdMapped FilePmdMapped Shared_Hugetlb Private_Hugetlb Swap SwapPss Locked THPeligible ProtectionKey VmFlags Mapping
555bc5570000 r--p 00000000 fd:00 402530 88 4 4 88 88 0 0 88 0 88 0 0 0 0 0 0 0 0 0 0 0 0 rd mr mw me sd unbound
555bc5586000 r-xp 00016000 fd:00 402530 924 4 4 724 724 0 0 724 0 724 0 0 0 0 0 0 0 0 0 0 0 0 rd ex mr mw me sd unbound
555bc566d000 r--p 000fd000 fd:00 402530 452 4 4 324 324 0 0 324 0 324 0 0 0 0 0 0 0 0 0 0 0 0 rd mr mw me sd unbound
555bc56df000 r--p 0016e000 fd:00 402530 8 4 4 8 8 0 0 0 8 8 8 0 0 0 0 0 0 0 0 0 0 0 rd mr mw me ac sd unbound
555bc56e1000 rw-p 00170000 fd:00 402530 44 4 4 44 44 0 0 0 44 44 44 0 0 0 0 0 0 0 0 0 0 0 rd wr mr mw me ac sd unbound
555bc56ec000 rw-p 00000000 00:00 0 4 4 4 4 4 0 0 0 4 4 4 0 0 0 0 0 0 0 0 0 0 0 rd wr mr mw me ac sd
555bc6ce7000 rw-p 00000000 00:00 0 8532104 4 4 1908712 1908712 0 0 0 1908712 1908712 1908712 0 1368064 0 0 0 0 0 0 0 1 0 rd wr mr mw me ac sd [heap]
7f327fb7e000 rw-p 00000000 00:00 0 1268 4 4 1268 1268 0 0 0 1268 1268 1268 0 0 0 0 0 0 0 0 0 0 0 rd wr mr mw me ac sd
======= ============== =========== ======= ======= ============ ============ ============= ============= ========== ========= ======== ============= ============== ============= ============== =============== ==== ======= ====== =========== =============
8552408 564 564 1919024 1914749 4856 0 2932 1911236 1919024 1911236 0 1368064 0 0 0 0 0 0 0 1 0 KB
The requirements to trigger this increase until kernel has no more memory to give:
- at least 5.x kernel
- unbound with large number of TCP sockets (e.g.: incoming-num-tcp: 1000 and do-tcp: yes)
- /sys/kernel/mm/transparent_hugepage/enabled has to be "always"
Debian 10 was by default with 4.19 kernel and we're safe, upgrading kernel to 5.10.127-2~bpo10 will trigger constant THP increasing. Fedora release 36 is running **5.18.**13-200.fc36.x86_64, but /sys/kernel/mm/transparent_hugepage/enabled = [madvise] prevented from THP increase. Changing THP to always causes THP constant increase.
[root@fedora ~]# cat /sys/kernel/mm/transparent_hugepage/enabled
always [madvise] never
[root@fedora ~]# echo always > /sys/kernel/mm/transparent_hugepage/enabled
[root@fedora ~]# cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never
[root@fedora ~]# sudo systemctl restart unbound
[root@fedora ~]# vmstat -aw 10
--procs-- -----------------------memory---------------------- ---swap-- -----io---- -system-- --------cpu--------
r b swpd free inact active si so bi bo in cs us sy id wa st
0 0 0 14280748 1480680 442792 0 0 0 1 129 154 0 0 100 0 0
0 0 0 14264364 1495844 442792 0 0 0 0 125 144 0 0 100 0 0
0 0 0 14247980 1511004 442792 0 0 0 13 130 142 0 0 100 0 0
0 0 0 14231596 1526172 442792 0 0 0 172 133 144 0 0 100 0 0
0 0 0 14215212 1541336 442792 0 0 0 0 132 145 0 0 100 0 0
0 0 0 14198828 1556504 442792 0 0 0 16 105 109 0 0 100 0 0
0 0 0 14182444 1571664 442792 0 0 0 0 96 98 0 0 100 0 0
Debian 11 (buster) and 12 (bookworm) are both with 5.x kernel and transparent_hugepage/enabled is "always". Result is continuous THP increasing until system RAM resource depletion.
To run unbound with TCP sockets on 5.x kernel safely: echo madvise >/sys/kernel/mm/transparent_hugepage/enabled
I have no idea, what are the counter effects of disabling transparent hugepages for unbound. Below is output from a live debian 10 (4.19 kernel) Unbound with large hot cache and 20K active TCP:853 sockets - 16.4GB of 24.3GB RES memory is used by transparent huge pages:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
unbound 29131 82.5 77.2 123220900 25451196 ? Ssl Jul23 62056:31 /usr/sbin/unbound -d -p -c /var/lib/unbound/etc/unbound/unbound.conf
$ top
MiB Mem : 32170.0 total, 5352.3 free, 25624.1 used, 1193.6 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5819.3 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
29131 unbound 20 0 117.5g 24.3g 7908 S 104.0 77.3 62057:11 unbound
$ sudo pmap -XX 29131
29131: /usr/sbin/unbound -d -p -c /var/lib/unbound/etc/unbound/unbound.conf
Address Perm Offset Device Inode Size KernelPageSize MMUPageSize Rss Pss Shared_Clean Shared_Dirty Private_Clean Private_Dirty Referenced Anonymous LazyFree AnonHugePages ShmemPmdMapped Shared_Hugetlb Private_Hugetlb Swap SwapPss Locked THPeligible ProtectionKey VmFlags Mapping
55d6b7985000 r--p 00000000 fd:00 928288 84 4 4 84 84 0 0 84 0 84 0 0 0 0 0 0 0 0 0 0 0 rd mr mw me dw sd unbound
55d6b799a000 r-xp 00015000 fd:00 928288 832 4 4 756 756 0 0 756 0 756 0 0 0 0 0 0 0 0 0 0 0 rd ex mr mw me dw sd unbound
55d6b7a6a000 r--p 000e5000 fd:00 928288 436 4 4 336 336 0 0 336 0 336 0 0 0 0 0 0 0 0 0 0 0 rd mr mw me dw sd unbound
55d6b7ad8000 r--p 00152000 fd:00 928288 8 4 4 8 8 0 0 0 8 8 8 0 0 0 0 0 0 0 0 0 0 rd mr mw me dw ac sd unbound
55d6b7ada000 rw-p 00154000 fd:00 928288 44 4 4 44 44 0 0 0 44 44 44 0 0 0 0 0 0 0 0 0 0 rd wr mr mw me dw ac sd unbound
55d6b8d6a000 rw-p 00000000 00:00 0 20389364 4 4 4168524 4168524 0 0 0 4168524 4168524 4168524 0 2871296 0 0 0 0 0 0 1 0 rd wr mr mw me ac sd [heap]
7f0c00000000 rw-p 00000000 00:00 0 1672 4 4 1564 1564 0 0 0 1564 1564 1564 0 0 0 0 0 0 0 0 1 0 rd wr mr mw me nr sd
7f0c001a2000 ---p 00000000 00:00 0 63864 4 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 mr mw me nr sd
7f0c08000000 rw-p 00000000 00:00 0 4496 4 4 4280 4280 0 0 0 4280 4280 4280 0 2048 0 0 0 0 0 0 1 0 rd wr mr mw me nr sd
7f0c08464000 ---p 00000000 00:00 0 61040 4 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 mr mw me nr sd
7f0c10000000 rw-p 00000000 00:00 0 10328 4 4 10256 10256 0 0 0 10256 10256 10256 0 6144 0 0 0 0 0 0 1 0 rd wr mr mw me nr sd
7f0c10a16000 ---p 00000000 00:00 0 55208 4 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 mr mw me nr sd
7f0c14000000 rw-p 00000000 00:00 0 13904 4 4 12504 12504 0 0 0 12504 12504 12504 0 6144 0 0 0 0 0 0 1 0 rd wr mr mw me nr sd
7f0c14d94000 ---p 00000000 00:00 0 51632 4 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 mr mw me nr sd
......................................................................................................................................................................................................
========= ============== =========== ======== ======== ============ ============ ============= ============= ========== ========= ======== ============= ============== ============== =============== ==== ======= ====== =========== =============
123220900 12900 12900 25452160 25447540 5684 0 2224 25444252 25452160 25444252 0 17281024 0 0 0 0 0 0 3106 0 KB
Is THP disabling or changing to madvise safe?