glances
glances copied to clipboard
nicolargo
possible memory leak
Check the bug Before filling this bug report, please search if a similar issue already exists. In this case, just add a comment on this existing issue.
Describe the bug A clear and concise description of what the bug is.
The memory usage reported by docker stats keeps increasing on 3 separated instances running on 3 different machines.
To Reproduce Steps to reproduce the behavior:
- Run the following command as root:
# docker compose up -d - add a line to the crontab of the root:
* * * * * /usr/bin/docker stats --no-stream --format '{{.MemUsage}}' glances | cut -d '/' -f 1 >>/var/log/docker_stats/glances.log
- Monitor the file
/var/log/docker_stats/glances.log
Expected behavior A clear and concise description of what you expected to happen.
I expect the memory logged in the glances.log to be stable over time.
Screenshots If applicable, add screenshots to help explain your problem.
Environement (please complete the following information)
- Operating System (lsb_release -a or OS name/version): Debian GNU/Linux 12 (bookworm)
- Glances & psutil versions:
Glances API version: 4
PsUtil version: 6.0.0
Log file: /tmp/glances-root.log
- How do you install Glances (Pypi package, script, package manager, source):
docker compose - Glances test:
===============================================================================
Glances 4.1.2 (/app/glances/__init__.py)
Python 3.11.9 (/venv/bin/python3)
PsUtil 6.0.0 (/venv/lib/python3.11/site-packages/psutil/__init__.py)
===============================================================================
alert [OK] 0.00003s
[]
amps [OK] 0.00023s key=name
[{'count': 0,
'countmax': None,
'countmin': 1.0,
'key': 'name',
'name': 'Dropbox',
'refresh': 3.0,
'regex': True,
'result': None,
'timer': 3.0309574604034424}, ...]
cloud [NA]
connections [NA]
containers [OK] 0.29408s key=name
[{'command': '/init',
'cpu_percent': 100.52314814814815,
'created': '2024-10-12T12:44:28.065151703Z',
'engine': 'docker',
'id': '9578831f53f31e32fb31428c91d871545fee2d0111ac184aebfe27c9a1e1ee29',
'image': ('linuxserver/bazarr:latest',),
'io_rx': None,
'io_wx': None,
'key': 'name',
'memory_percent': None,
'memory_usage': 323547136,
'name': 'bazarr',
'network_rx': None,
'network_tx': None,
'status': 'running',
'uptime': 'yesterday'}, ...]
core [OK] 0.00713s
{'log': 48, 'phys': 24}
cpu [OK] 0.00180s
{'cpucore': 48,
'ctx_switches': 0,
'ctx_switches_gauge': 3156258973,
'guest': 0.0,
'idle': 96.2,
'interrupts': 0,
'interrupts_gauge': 1395350189,
'iowait': 0.0,
'irq': 0.0,
'nice': 0.0,
'soft_interrupts': 0,
'soft_interrupts_gauge': 1355389962,
'steal': 0.0,
'syscalls': 0,
'syscalls_gauge': 0,
'system': 1.9,
'time_since_update': 2.3210175037384033,
'total': 4.2,
'user': 1.8}
diskio [OK] 0.00596s key=disk_name
[{'disk_name': 'nvme0n1',
'key': 'disk_name',
'read_bytes': 0,
'read_bytes_gauge': 19937036288,
'read_count': 0,
'read_count_gauge': 919639,
'time_since_update': 2.326836109161377,
'write_bytes': 0,
'write_bytes_gauge': 50798579712,
'write_count': 0,
'write_count_gauge': 4789946}, ...]
folders [OK] 0.00005s
[]
fs [OK] 0.00130s key=mnt_point
[{'device_name': 'tank/docker',
'free': 125379863248896,
'fs_type': 'zfs',
'key': 'mnt_point',
'mnt_point': '/config',
'percent': 0.0,
'size': 125394844778496,
'used': 14981529600}, ...]
gpu [OK] 0.00004s
[]
help [OK] 0.00001s
None
ip [OK] 0.00233s
{'address': '***',
'gateway': '***',
'mask': '***',
'mask_cidr': '***',
'public_address': '***',
'public_info_human': '***'}
irq [NA]
load [OK] 0.00006s
{'cpucore': 48, 'min1': 2.109375, 'min15': 1.20703125, 'min5': 1.46728515625}
mem [OK] 0.00028s
{'active': 870154240,
'available': 61882241024,
'buffers': 15302656,
'cached': 2608340992,
'free': 61882241024,
'inactive': 8887496704,
'percent': 77.1,
'shared': 180944896,
'total': 270106365952,
'used': 208224124928}
memswap [OK] 0.00052s
{'free': 0, 'percent': 0.0, 'sin': 0, 'sout': 0, 'time_since_update': 2.337296962738037, 'total': 0, 'used': 0}
network [OK] 0.00664s key=interface_name
[{'alias': None,
'bytes_all': 200,
'bytes_all_gauge': 6824554,
'bytes_all_rate_per_sec': 86.0,
'bytes_recv': 100,
'bytes_recv_gauge': 3412277,
'bytes_recv_rate_per_sec': 43.0,
'bytes_sent': 100,
'bytes_sent_gauge': 3412277,
'bytes_sent_rate_per_sec': 43.0,
'interface_name': 'lo',
'key': 'interface_name',
'speed': 0,
'time_since_update': 2.3232762813568115}, ...]
now [OK] 0.00006s
{'custom': '2024-10-13 22:38:44 BST', 'iso': '2024-10-13T22:38:44+01:00'}
percpu [OK] 0.00266s key=cpu_number
[{'cpu_number': 0,
'dpc': None,
'guest': 0.0,
'guest_nice': 0.0,
'idle': 96.4,
'interrupt': None,
'iowait': 1.1,
'irq': 0.0,
'key': 'cpu_number',
'nice': 0.7,
'softirq': 0.0,
'steal': 0.0,
'system': 0.4,
'total': 3.6,
'user': 1.4}, ...]
ports [OK] 0.00001s
[]
processcount [OK] 0.12034s
{'pid_max': 0, 'running': 1, 'sleeping': 288, 'thread': 1826, 'total': 289}
processlist [OK] 0.00108s
[]
psutilversion [OK] 0.00017s
'6.0.0'
quicklook [OK] 0.00151s
{'cpu': 4.2,
'cpu_hz': 3000000000.0,
'cpu_hz_current': 859181958.3333333,
'cpu_log_core': 48,
'cpu_name': 'Intel(R) Xeon(R) Silver 4116 CPU @ 2.10GHz',
'cpu_phys_core': 24,
'load': 2.5,
'mem': 77.1,
'percpu': [{...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...},
{...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...},
{...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...},
{...}, {...}, {...}],
'swap': 0.0}
raid [NA]
sensors [OK] 0.00008s key=label
[{'critical': 79,
'key': 'label',
'label': 'Composite',
'type': <SensorType.CPU_TEMP: 'temperature_core'>,
'unit': 'C',
'value': 38,
'warning': 74}, ...]
smart [NA]
system [OK] 0.00002s
{'hostname': 'rocker',
'hr_name': 'Alpine Linux 3.19.2 64bit / Linux 6.6.32-production+truenas',
'linux_distro': 'Alpine Linux 3.19.2',
'os_name': 'Linux',
'os_version': '6.6.32-production+truenas',
'platform': '64bit'}
uptime [OK] 0.00044s
{'seconds': 284581}
version [OK] 0.00013s
'4.1.2'
wifi [OK] 0.00008s
[]
===============================================================================
Total time to update all stats: 0.45255s
===============================================================================
Additional context Add any other context about the problem here.
Here is a sample of the content of the file glances.log after 2 days of running:
112.4MiB
113.4MiB
113.4MiB
113.4MiB
113.4MiB
113.1MiB
113MiB
113.5MiB
113.5MiB
113.5MiB
113.5MiB
113.5MiB
113.4MiB
* snip … snip *
193.8MiB
194MiB
194MiB
194MiB
194.1MiB
194.2MiB
194.3MiB
194.3MiB
194.4MiB
194.5MiB
194.7MiB
194.6MiB
194.6MiB
194.6MiB
194.6MiB
194.8MiB
the content of the compose.yaml:
---
#version: "3"
services:
glances:
image: nicolargo/glances:latest-full
container_name: glances
privileged: true
deploy:
resources:
limits:
memory: 512M
pid: host
network_mode: host
environment:
- "TZ=Europe/London"
- "GLANCES_OPT=-C /config/glances.conf -w --password -u ha"
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- ./config:/config:ro
secrets:
- source: ha_password
target: /root/.config/glances/ha.pwd
restart: unless-stopped
secrets:
ha_password:
file: ./secrets/ha_password.txt
modification of glances.conf:
[gpu]
disable=True
[hddtemp]
disable=True
[graph]
generate_every=0
[diskio]
hide=loop.*,/dev/loop.*,sd*
[fs]
hide=/boot.*,/snap.*,/etc/*,_config/*
You can also pastebin:
- the Glances configuration file (https://glances.readthedocs.io/en/latest/config.html#location)
- the Glances log file (https://glances.readthedocs.io/en/latest/config.html#logging)
Hi @janusn
in the Glances configuration file you have the following key:
[global]
# History size (maximum number of values)
# Default is 1200 values (~1h with the default refresh rate)
history_size=1200
So it's a "normal" behavior to have your memory increase for the first hour.
After that it should be stable.
I have been monitoring the memory consumption since your reply. There were a few set backs such as container recreated by watchtower. Nonetheless, here is what I have found.
The memory usage on 2 of 3 instances of the glances keep growing over 24 hours period but the other one does not. Here is a summary of memory usage of one of the growing instances:
73.51MiB (runtime 00:00 hours)
104.9MiB (runtime 01:00 hour)
105.3MiB (runtime 02:00 hours)
107.7MiB (runtime 03:00 hours)
108.3MiB (runtime 04:00 hours)
109.8MiB (runtime 05:00 hours)
110MiB (runtime 06:00 hours)
112.3MiB (runtime 07:00 hours)
113.9MiB (runtime 08:00 hours)
114.7MiB (runtime 09:00 hours)
115MiB (runtime 10:00 hours)
117.1MiB (runtime 11:00 hours)
116.1MiB (runtime 12:00 hours)
117MiB (runtime 13:00 hours)
117.3MiB (runtime 14:00 hours)
119.1MiB (runtime 15:00 hours)
120.7MiB (runtime 16:00 hours)
121.5MiB (runtime 17:00 hours)
121.1MiB (runtime 18:00 hours)
121.7MiB (runtime 19:00 hours)
122.2MiB (runtime 20:00 hours)
122.6MiB (runtime 21:00 hours)
124.8MiB (runtime 22:00 hours)
124.5MiB (runtime 23:00 hours)
126MiB (runtime 24:00 hours)
126.2MiB (runtime 25:00 hours)
125.4MiB (runtime 26:00 hours)
126.3MiB (runtime 27:00 hours)
127.4MiB (runtime 28:00 hours)
127.6MiB (runtime 29:00 hours)
129.7MiB (runtime 30:00 hours)
130.4MiB (runtime 31:00 hours)
130.4MiB (runtime 32:00 hours)
132.2MiB (runtime 33:00 hours)
On my configuration (running Glances in local outside a Docker container) and with a history_size=3.
When Glances start:
rss=64245760, vms=590016512, shared=17039360, text=3026944, lib=0, data=91455488, dirty=0, uss=51081216, pss=52712448, swap=0
One hour later:
rss=65421312, vms=590155776, shared=17039360, text=3026944, lib=0, data=91594752, dirty=0, uss=51511296, pss=53140480, swap=0
So memory increase around 1.4 MB (regarding 31 MB on your test).
I will make a long term test next week.
Also reproduced on my side (in console mode with default history_size=1200) during 30 hours but with a more limited leak:
Test with history_size=0 (in red):
Ok found it thanks to mem_top:
2024-10-31 08:40:16,583 -- INFO --
refs:
1519 <class 'dict'> {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
1386 <class 'dict'> {140455405909952: <weakref at 0x7fbe529dfdb0; to 'type' at 0x7fbe529377c0 (type)>, 140455405907776:
870 <class 'dict'> {1018938: {'cmdline': ['/home/nhe/bin/idea-IC-233.15026.9/jbr/bin/java', '-classpath', '/home/nhe/bi
870 <class 'dict'> {1: psutil.Process(pid=1, name='systemd', status='sleeping', started='2024-06-24 10:19:03'), 2: psut
870 <class 'dict'> {3119283: {'cpu_percent': {'decoration': 'DEFAULT', 'optional': False, 'additional': False, 'splitta
756 <class 'dict'> {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib':
523 <class 'collections.defaultdict'> defaultdict(<class 'int'>, {('sda', 0): 0, ('sda', 1): 0, ('sda', 2): 0, ('sda', 3): 0, ('sda', 4):
450 <class 'list'> [(0, 0, 0), (1, 31, -1), (127, 159, -1), (173, 173, 0), (768, 879, 0), (1155, 1161, 0), (1425, 1469,
435 <class 'list'> [{'cpu_percent': 12.2, 'num_threads': 1, 'name': 'python', 'memory_percent': 0.19223070159317707, 'g
423 <class 'dict'> {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions
bytes:
73816 {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
36960 {140455405909952: <weakref at 0x7fbe529dfdb0; to 'type' at 0x7fbe529377c0 (type)>, 140455405907776:
36960 {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib':
18520 {'__name__': 'os', '__doc__': "OS routines for NT or Posix depending on what system we're on.\n\nThi
18520 {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions
18520 {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520 {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520 {'__name__': 'curses', '__doc__': 'curses\n\nThe main package for curses support for Python. Normal
18520 {'__name__': '_curses', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_externa
18520 {1018938: {'cmdline': ['/home/nhe/bin/idea-IC-233.15026.9/jbr/bin/java', '-classpath', '/home/nhe/bi
types:
11694 <class 'function'>
9483 <class 'dict'>
7183 <class 'tuple'>
2653 <class 'weakref'>
2237 <class 'list'>
2050 <class 'cell'>
2003 <class 'getset_descriptor'>
1574 <class 'type'>
1394 <class 'wrapper_descriptor'>
1339 <class 'builtin_function_or_method'>
===
2024-10-31 09:26:12,157 -- INFO --
refs:
1690 <class 'dict'> {1018938: {'cmdline': ['/home/nhe/bin/idea-IC-233.15026.9/jbr/bin/java', '-classpath', '/home/nhe/bi
1519 <class 'dict'> {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
1386 <class 'dict'> {140455405909952: <weakref at 0x7fbe529dfdb0; to 'type' at 0x7fbe529377c0 (type)>, 140455405907776:
1200 <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 40, 43, 551316), 2.0), (datetime.datetime(2024, 10, 31, 8, 40,
1200 <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 40, 43, 551311), 7.1), (datetime.datetime(2024, 10, 31, 8, 40,
1200 <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 40, 43, 552876), 64.8), (datetime.datetime(2024, 10, 31, 8, 40,
1200 <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 40, 43, 552878), 84.9), (datetime.datetime(2024, 10, 31, 8, 40,
1200 <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 40, 43, 553628), 0.74), (datetime.datetime(2024, 10, 31, 8, 40,
1200 <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 40, 43, 553631), 0.64), (datetime.datetime(2024, 10, 31, 8, 40,
1200 <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 40, 43, 556058), 1.3), (datetime.datetime(2024, 10, 31, 8, 40,
bytes:
73816 {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
36960 {140455405909952: <weakref at 0x7fbe529dfdb0; to 'type' at 0x7fbe529377c0 (type)>, 140455405907776:
36960 {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib':
36960 {1018938: {'cmdline': ['/home/nhe/bin/idea-IC-233.15026.9/jbr/bin/java', '-classpath', '/home/nhe/bi
36960 {1: psutil.Process(pid=1, name='systemd', status='sleeping', started='2024-06-24 10:19:03'), 2: psut
18520 {'__name__': 'os', '__doc__': "OS routines for NT or Posix depending on what system we're on.\n\nThi
18520 {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions
18520 {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520 {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520 {'__name__': 'curses', '__doc__': 'curses\n\nThe main package for curses support for Python. Normal
types:
11694 <class 'function'>
9905 <class 'dict'>
8377 <class 'tuple'>
3985 <class 'list'>
2651 <class 'weakref'>
2050 <class 'cell'>
2003 <class 'getset_descriptor'>
1574 <class 'type'>
1394 <class 'wrapper_descriptor'>
1338 <class 'builtin_function_or_method'>
===
AFTER a F5 (clear processlist cache)
2024-10-31 09:31:11,215 -- INFO --
refs:
1519 <class 'dict'> {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
1386 <class 'dict'> {140455405909952: <weakref at 0x7fbe529dfdb0; to 'type' at 0x7fbe529377c0 (type)>, 140455405907776:
1200 <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 45, 41, 456881), 0.5), (datetime.datetime(2024, 10, 31, 8, 45,
1200 <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 45, 41, 456876), 2.4), (datetime.datetime(2024, 10, 31, 8, 45,
1200 <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 45, 41, 458499), 64.8), (datetime.datetime(2024, 10, 31, 8, 45,
1200 <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 45, 41, 458501), 84.9), (datetime.datetime(2024, 10, 31, 8, 45,
1200 <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 45, 41, 459220), 0.52), (datetime.datetime(2024, 10, 31, 8, 45,
1200 <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 45, 41, 459222), 0.56), (datetime.datetime(2024, 10, 31, 8, 45,
1200 <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 45, 41, 461633), 1.3), (datetime.datetime(2024, 10, 31, 8, 45,
1200 <class 'list'> [(datetime.datetime(2024, 10, 31, 8, 45, 41, 461635), 0.4), (datetime.datetime(2024, 10, 31, 8, 45,
bytes:
73816 {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
36960 {140455405909952: <weakref at 0x7fbe529dfdb0; to 'type' at 0x7fbe529377c0 (type)>, 140455405907776:
36960 {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib':
18520 {'__name__': 'os', '__doc__': "OS routines for NT or Posix depending on what system we're on.\n\nThi
18520 {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions
18520 {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520 {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520 {'__name__': 'curses', '__doc__': 'curses\n\nThe main package for curses support for Python. Normal
18520 {'__name__': '_curses', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_externa
18520 {1018938: {'cmdline': ['/home/nhe/bin/idea-IC-233.15026.9/jbr/bin/java', '-classpath', '/home/nhe/bi
types:
11694 <class 'function'>
9433 <class 'dict'>
8377 <class 'tuple'>
3552 <class 'list'>
2651 <class 'weakref'>
2050 <class 'cell'>
2003 <class 'getset_descriptor'>
1574 <class 'type'>
1394 <class 'wrapper_descriptor'>
1338 <class 'builtin_function_or_method'>
The issue is confirmed and is in the processes.py file / class GlancesProcesses / dict self.processlist_cache.
The size of this dict can be very large because the key is the PID, on Linux system the default value for this file, 32768, results in the same range of PIDs as on earlier kernels. On 32-bit platforms, 32768 is the maximum value for pid_max. On 64-bit systems, pid_max can be set to any value up to 2^22 (PID_MAX_LIMIT, approximately 4 million)...
So we need to have a peace of code after the main loop of the update method to clean key/value when PID is no more existing:
- during the loop: build the list of current PIDs or use the processlist variable (list of dict)
- after the loop, remove from the self.processlist_cache key/value if key is not in the list of current PIDs
Adding this code after the loop:
# Remove non running process from the cache (avoid issue #2976)
pids_running = [p['pid'] for p in processlist]
pids_cached = [p for p in self.processlist_cache.keys()]
for pid in pids_cached:
if pid not in pids_running:
self.processlist_cache.pop(pid, None)
seems to solve the leak:
Running Glances with (history_size=0):
$ while true; do ./venv/bin/python -c 'import os; import time; import psutil; g = psutil.Process(3223060); print("{} {}".format(time.strftime("%Y-%m-%d %H:%M:%S"), g.memory_full_info()));'; sleep 300; done
2024-10-31 16:10:03 pfullmem(rss=70799360, vms=1403809792, shared=15728640, text=4096, lib=0, data=152330240, dirty=0, uss=60723200, pss=63950848, swap=0)
2024-10-31 16:15:03 pfullmem(rss=70557696, vms=1403809792, shared=15728640, text=4096, lib=0, data=152330240, dirty=0, uss=60678144, pss=63902720, swap=0)
2024-10-31 16:20:03 pfullmem(rss=72482816, vms=1405534208, shared=15728640, text=4096, lib=0, data=154054656, dirty=0, uss=62652416, pss=65876992, swap=0)
2024-10-31 16:25:03 pfullmem(rss=71524352, vms=1404329984, shared=15728640, text=4096, lib=0, data=152850432, dirty=0, uss=61501440, pss=64351232, swap=0)
2024-10-31 16:30:03 pfullmem(rss=71847936, vms=1404592128, shared=15728640, text=4096, lib=0, data=153120768, dirty=0, uss=61755392, pss=64605184, swap=0)
Also confirmed by mem_top:
2024-10-31 16:07:57,006 -- INFO --
refs:
1519 <class 'dict'> {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
1386 <class 'dict'> {140140222838720: <weakref at 0x7f74f0399db0; to 'type' at 0x7f74f03377c0 (type)>, 140140222836544:
890 <class 'dict'> {1018938: {'cmdline': ['/home/nhe/bin/idea-IC-233.15026.9/jbr/bin/java', '-classpath', '/home/nhe/bi
890 <class 'dict'> {1: psutil.Process(pid=1, name='systemd', status='sleeping', started='2024-06-24 10:19:03'), 2: psut
890 <class 'dict'> {16089: {'num_threads': {'decoration': 'DEFAULT', 'optional': False, 'additional': False, 'splittabl
756 <class 'dict'> {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib':
523 <class 'collections.defaultdict'> defaultdict(<class 'int'>, {('sda', 0): 0, ('sda', 1): 0, ('sda', 2): 0, ('sda', 3): 0, ('sda', 4):
450 <class 'list'> [(0, 0, 0), (1, 31, -1), (127, 159, -1), (173, 173, 0), (768, 879, 0), (1155, 1161, 0), (1425, 1469,
445 <class 'list'> [{'num_threads': 26, 'pid': 16089, 'memory_percent': 2.455672030884629, 'cpu_percent': 19.7, 'gids':
423 <class 'dict'> {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions
bytes:
73816 {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
36960 {140140222838720: <weakref at 0x7f74f0399db0; to 'type' at 0x7f74f03377c0 (type)>, 140140222836544:
36960 {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib':
18520 {'__name__': 'os', '__doc__': "OS routines for NT or Posix depending on what system we're on.\n\nThi
18520 {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions
18520 {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520 {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520 {'__name__': 'curses', '__doc__': 'curses\n\nThe main package for curses support for Python. Normal
18520 {'__name__': '_curses', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_externa
18520 {1018938: {'cmdline': ['/home/nhe/bin/idea-IC-233.15026.9/jbr/bin/java', '-classpath', '/home/nhe/bi
types:
11701 <class 'function'>
10050 <class 'dict'>
7218 <class 'tuple'>
2693 <class 'weakref'>
2584 <class 'list'>
2090 <class 'cell'>
2003 <class 'getset_descriptor'>
1574 <class 'type'>
1514 <class 'builtin_function_or_method'>
1394 <class 'wrapper_descriptor'>
===
2024-10-31 16:34:09,533 -- INFO --
refs:
1519 <class 'dict'> {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
1386 <class 'dict'> {140140222838720: <weakref at 0x7f74f0399db0; to 'type' at 0x7f74f03377c0 (type)>, 140140222836544:
926 <class 'dict'> {1018938: {'cmdline': ['/home/nhe/bin/idea-IC-233.15026.9/jbr/bin/java', '-classpath', '/home/nhe/bi
926 <class 'dict'> {1: psutil.Process(pid=1, name='systemd', status='sleeping', started='2024-06-24 10:19:03'), 2: psut
926 <class 'dict'> {3223060: {'num_threads': {'decoration': 'DEFAULT', 'optional': False, 'additional': False, 'splitta
756 <class 'dict'> {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib':
523 <class 'collections.defaultdict'> defaultdict(<class 'int'>, {('sda', 0): 0, ('sda', 1): 0, ('sda', 2): 0, ('sda', 3): 0, ('sda', 4):
463 <class 'list'> [{'num_threads': 9, 'pid': 3223060, 'memory_percent': 0.21623942888930556, 'cpu_percent': 14.1, 'gid
450 <class 'list'> [(0, 0, 0), (1, 31, -1), (127, 159, -1), (173, 173, 0), (768, 879, 0), (1155, 1161, 0), (1425, 1469,
423 <class 'dict'> {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions
bytes:
73816 {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
36960 {140140222838720: <weakref at 0x7f74f0399db0; to 'type' at 0x7f74f03377c0 (type)>, 140140222836544:
36960 {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib':
36960 {1: psutil.Process(pid=1, name='systemd', status='sleeping', started='2024-06-24 10:19:03'), 2: psut
18520 {'__name__': 'os', '__doc__': "OS routines for NT or Posix depending on what system we're on.\n\nThi
18520 {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions
18520 {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520 {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520 {'__name__': 'curses', '__doc__': 'curses\n\nThe main package for curses support for Python. Normal
18520 {'__name__': '_curses', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_externa
types:
11701 <class 'function'>
10147 <class 'dict'>
7218 <class 'tuple'>
2698 <class 'list'>
2693 <class 'weakref'>
2090 <class 'cell'>
2003 <class 'getset_descriptor'>
1574 <class 'type'>
1514 <class 'builtin_function_or_method'>
1394 <class 'wrapper_descriptor'>
2024-10-31 16:34:11,742 -- INFO --
refs:
1519 <class 'dict'> {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
1386 <class 'dict'> {140140222838720: <weakref at 0x7f74f0399db0; to 'type' at 0x7f74f03377c0 (type)>, 140140222836544:
926 <class 'dict'> {1018938: {'cmdline': ['/home/nhe/bin/idea-IC-233.15026.9/jbr/bin/java', '-classpath', '/home/nhe/bi
926 <class 'dict'> {1: psutil.Process(pid=1, name='systemd', status='sleeping', started='2024-06-24 10:19:03'), 2: psut
926 <class 'dict'> {2175857: {'num_threads': {'decoration': 'DEFAULT', 'optional': False, 'additional': False, 'splitta
756 <class 'dict'> {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib':
523 <class 'collections.defaultdict'> defaultdict(<class 'int'>, {('sda', 0): 0, ('sda', 1): 0, ('sda', 2): 0, ('sda', 3): 0, ('sda', 4):
463 <class 'list'> [{'num_threads': 22, 'pid': 2175857, 'memory_percent': 10.093652599903841, 'cpu_percent': 151.6, 'gi
450 <class 'list'> [(0, 0, 0), (1, 31, -1), (127, 159, -1), (173, 173, 0), (768, 879, 0), (1155, 1161, 0), (1425, 1469,
423 <class 'dict'> {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions
bytes:
73816 {'__name__': 'pynvml', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_external
36960 {140140222838720: <weakref at 0x7f74f0399db0; to 'type' at 0x7f74f03377c0 (type)>, 140140222836544:
36960 {'sys': <module 'sys' (built-in)>, 'builtins': <module 'builtins' (built-in)>, '_frozen_importlib':
36960 {1: psutil.Process(pid=1, name='systemd', status='sleeping', started='2024-06-24 10:19:03'), 2: psut
18520 {'__name__': 'os', '__doc__': "OS routines for NT or Posix depending on what system we're on.\n\nThi
18520 {'__name__': 'socket', '__doc__': "This module provides socket operations and some related functions
18520 {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520 {'__name__': '_socket', '__doc__': 'Implementation module for socket operations.\n\nSee the socket m
18520 {'__name__': 'curses', '__doc__': 'curses\n\nThe main package for curses support for Python. Normal
18520 {'__name__': '_curses', '__doc__': None, '__package__': '', '__loader__': <_frozen_importlib_externa
types:
11701 <class 'function'>
10148 <class 'dict'>
7218 <class 'tuple'>
2699 <class 'list'>
2693 <class 'weakref'>
2090 <class 'cell'>
2003 <class 'getset_descriptor'>
1574 <class 'type'>
1514 <class 'builtin_function_or_method'>
1394 <class 'wrapper_descriptor'>
had to make some extra-test before pushing but look ok.
Ok pushed on the develop branch.
After some test, il will release a version 4.2.1.
@nicolargo
Update:
I checked again and I realized that the image tagged 4.2.1-full is different from the image I was running, tagged latest-full. Let me run the 4.2.1-full a couple day. I will report my result afterwards
$ sudo docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
nicolargo/glances 4.2.1-full a03e9d74b587 3 days ago 197MB
nicolargo/glances latest-full d9f6a9532303 3 days ago 197MB
Original comment:
Thanks for the quick fix.
On one of the 3 containers, the memory usage is still growing albeit it is much slower than before.
11/2
05:17:03 65.92MiB (container restarted by watchtower)
06:00:03 70.41MiB
07:00:03 79.04MiB
08:00:03 96.26MiB
09:00:03 98.54MiB
10:00:03 97.65MiB
11:00:03 97.6MiB
12:00:04 98.7MiB
13:00:03 98.39MiB
14:00:03 99.37MiB
15:00:03 99.45MiB
16:00:03 99.29MiB
17:00:04 98.3MiB
18:00:03 99.8MiB
19:00:03 99.16MiB
20:00:03 100.5MiB
21:00:03 99.81MiB
22:00:03 99.62MiB
23:00:03 99.59MiB
11/3
00:00:04 99.93MiB
01:00:04 99.63MiB
02:00:03 99.88MiB
03:00:04 99.86MiB
04:00:03 100MiB
05:00:03 100.1MiB
06:00:03 101.7MiB
07:00:03 100.5MiB
08:00:03 100.7MiB
09:00:04 100.7MiB
10:00:03 101.4MiB
11:00:03 101.4MiB
12:00:04 101.4MiB
13:00:04 101.4MiB
14:00:04 101.4MiB
15:00:03 104.2MiB
16:00:03 104.3MiB
17:00:04 104.5MiB
18:00:04 104.4MiB
19:00:03 103.5MiB
20:00:02 104.7MiB
21:00:03 104.4MiB
22:00:03 104.4MiB
23:00:03 104.6MiB
11/4
00:00:03 103.5MiB
01:00:04 103.6MiB
02:00:03 105.2MiB
03:00:03 105.3MiB
04:00:03 105.1MiB
05:00:03 104.2MiB
06:00:03 105.3MiB
07:00:03 104.3MiB
08:01:04 104.2MiB
09:00:03 105.5MiB
10:00:04 104.5MiB
11:00:04 104.5MiB
12:00:03 105.7MiB
13:00:04 105.4MiB
14:00:03 105.6MiB
15:00:03 105.4MiB
16:00:03 106.2MiB
17:00:03 106.1MiB
18:00:03 105.7MiB
19:00:03 106.2MiB
20:00:03 106.2MiB
21:00:03 106.4MiB
22:00:03 106.4MiB
23:00:04 107.7MiB
11/5
00:00:03 108.1MiB
Unfortunately, 4.2.1-full still exhibits this memory leak. I have included logs from the three install starting from restarting of the containers. logs.zip
I still see this leak on 4.3.2 (running as a container). The drops signify manual container restarts.
What's your Glances configuration and command line ?
config:
[prometheus]
host=192.168.0.101
port=9091
labels=hostname:ubuntu-laptop
command:
GLANCES_OPT=--export prometheus -C /glances/conf/glances.conf --quiet
@DeepSpace2 can you try with the following options:
GLANCES_OPT=--export prometheus -C /glances/conf/glances.conf --disable-history --quiet
@DeepSpace2 can you try with the following options:
GLANCES_OPT=--export prometheus -C /glances/conf/glances.conf --disable-history --quiet
@nicolargo
--disable-history made a huge difference. Memory is stable on ~80MB instead of slowly creeping to 1GB.