Memory leak in dll mode accumulating when switching maps

[illwieckz]

Reported by mail by bmorel.

Reproduction:

• system: debian stable amd64
• compile with dll mode, RelWithDebInfo
• disable overcommit (to get crashes earlier and avoid thrashing on linux)
• add "nextmap" in config/map/default.cfg
• run daemonded in dll mode and watch it's memory usage, up to "out of memory" signals
• the engine never shuts down, and thus a coredump is never created

What should happen:

• memory usage should not increase for every map loaded or game started
• daemonded should close instead of going inactive for hours, so that a supervision may do it's job of mitigating crash consequences
• a coredump should be generated

Example of logs (including system ones) when the situation happens:

dmesg:

2025-02-06T11:37:42,661193+00:00 __vm_enough_memory: pid: 15774, comm: daemonded, no enough memory for the allocation
2025-02-06T11:37:42,661224+00:00 __vm_enough_memory: pid: 15774, comm: daemonded, no enough memory for the allocation
2025-02-06T11:37:45,162076+00:00 __vm_enough_memory: pid: 15774, comm: daemonded, no enough memory for the allocation
2025-02-06T11:37:45,162087+00:00 __vm_enough_memory: pid: 15774, comm: daemonded, no enough memory for the allocation
2025-02-06T11:37:45,162089+00:00 __vm_enough_memory: pid: 15774, comm: daemonded, no enough memory for the allocation
2025-02-06T11:37:45,162090+00:00 __vm_enough_memory: pid: 15774, comm: daemonded, no enough memory for the allocation
2025-02-06T11:37:45,162092+00:00 __vm_enough_memory: pid: 15774, comm: daemonded, no enough memory for the allocation
2025-02-06T11:37:45,162093+00:00 __vm_enough_memory: pid: 15774, comm: daemonded, no enough memory for the allocation
2025-02-06T11:37:45,162095+00:00 __vm_enough_memory: pid: 15774, comm: daemonded, no enough memory for the allocation
2025-02-06T11:37:45,162096+00:00 __vm_enough_memory: pid: 15774, comm: daemonded, no enough memory for the allocation
2025-02-10T10:08:07,791785+00:00 __vm_enough_memory: 792 callbacks suppressed
2025-02-10T10:08:07,791795+00:00 __vm_enough_memory: pid: 12441, comm: daemonded, no enough memory for the allocation
2025-02-10T10:08:07,791809+00:00 __vm_enough_memory: pid: 12441, comm: daemonded, no enough memory for the allocation
2025-02-10T10:08:07,802162+00:00 __vm_enough_memory: pid: 12441, comm: daemonded, no enough memory for the allocation
2025-02-10T10:08:07,802179+00:00 __vm_enough_memory: pid: 12441, comm: daemonded, no enough memory for the allocation
2025-02-10T10:08:07,802198+00:00 __vm_enough_memory: pid: 12441, comm: daemonded, no enough memory for the allocation
2025-02-10T10:08:07,802201+00:00 __vm_enough_memory: pid: 12441, comm: daemonded, no enough memory for the allocation
2025-02-10T10:08:07,802203+00:00 __vm_enough_memory: pid: 12441, comm: daemonded, no enough memory for the allocation
2025-02-10T11:49:45,415094+00:00 __vm_enough_memory: pid: 25671, comm: nacl_helper_boo, no enough memory for the allocation
2025-02-10T11:49:45,552329+00:00 __vm_enough_memory: pid: 25673, comm: nacl_helper_boo, no enough memory for the allocation
2025-02-10T11:50:29,757170+00:00 __vm_enough_memory: pid: 25783, comm: nacl_helper_boo, no enough memory for the allocation
2025-02-10T11:50:29,878847+00:00 __vm_enough_memory: pid: 25785, comm: nacl_helper_boo, no enough memory for the allocation
2025-02-10T11:57:35,237775+00:00 __vm_enough_memory: pid: 26711, comm: nacl_helper_boo, no enough memory for the allocation
2025-02-10T11:57:35,376574+00:00 __vm_enough_memory: pid: 26713, comm: nacl_helper_boo, no enough memory for the allocation
2025-02-11T22:20:16,283992+00:00 __vm_enough_memory: pid: 26546, comm: nacl_helper_boo, no enough memory for the allocation
2025-02-13T10:28:12,981957+00:00 __vm_enough_memory: pid: 13137, comm: daemonded, no enough memory for the allocation
2025-02-13T10:28:12,981980+00:00 __vm_enough_memory: pid: 13137, comm: daemonded, no enough memory for the allocation
2025-02-13T10:28:12,981996+00:00 __vm_enough_memory: pid: 13137, comm: daemonded, no enough memory for the allocation
2025-02-14T07:50:41,931662+00:00 __vm_enough_memory: pid: 27524, comm: daemonded, no enough memory for the allocation
2025-02-14T07:50:41,931675+00:00 __vm_enough_memory: pid: 27524, comm: daemonded, no enough memory for the allocation
2025-02-14T07:50:41,931677+00:00 __vm_enough_memory: pid: 27524, comm: daemonded, no enough memory for the allocation
2025-02-14T07:50:41,931683+00:00 __vm_enough_memory: pid: 27524, comm: daemonded, no enough memory for the allocation
2025-02-14T07:50:41,968643+00:00 __vm_enough_memory: pid: 13909, comm: daemonded, no enough memory for the allocation
2025-02-14T07:50:41,968654+00:00 __vm_enough_memory: pid: 13909, comm: daemonded, no enough memory for the allocation
2025-02-14T07:50:41,978680+00:00 __vm_enough_memory: pid: 13909, comm: daemonded, no enough memory for the allocation
2025-02-14T07:50:41,978696+00:00 __vm_enough_memory: pid: 13909, comm: daemonded, no enough memory for the allocation
2025-02-14T07:50:41,978708+00:00 __vm_enough_memory: pid: 13909, comm: daemonded, no enough memory for the allocation
2025-02-14T07:50:41,978711+00:00 __vm_enough_memory: pid: 13909, comm: daemonded, no enough memory for the allocation

stdout under log-rotate:

2025-02-14_07:50:42.39735 ClientDisconnect: 12 [] (XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX) "^9[bot]^4 Edwin M. Lillibridge^*"
2025-02-14_07:50:42.39939 Waiting for the VM thread...
2025-02-14_07:50:42.40470 ----- Server Initialization -----
2025-02-14_07:50:42.40475 Map: metro-b1-2
2025-02-14_07:50:42.42101 [FS] Loading pak '/home/user/www/unvanquished/mods/deadbeef/mod-deadbeef-light_0.55.0-17.dpk'...
2025-02-14_07:50:42.42141 [FS] Loading pak '/home/user/unvanquished/pkg-release/unvanquished_0.55.1.dpk'...
2025-02-14_07:50:42.42243 [FS] Loading pak '/home/user/unvanquished/pkg-release/tex-common_2.4.dpk'...
2025-02-14_07:50:42.42270 [FS] Loading pak '/home/user/unvanquished/pkg-release/res-players_0.55.dpk'...
2025-02-14_07:50:42.42318 [FS] Loading pak '/home/user/unvanquished/pkg-release/res-legacy_0.55.dpk'...
2025-02-14_07:50:42.42385 [FS] Loading pak '/home/user/unvanquished/pkg-release/res-weapons_0.55.dpk'...
2025-02-14_07:50:42.42480 [FS] Loading pak '/home/user/unvanquished/pkg-release/res-buildables_0.55.dpk'...
2025-02-14_07:50:42.42543 [FS] Loading pak '/home/user/unvanquished/pkg-release/res-voices_0.54.dpk'...
2025-02-14_07:50:42.42581 [FS] Loading pak '/home/user/unvanquished/pkg-release/res-soundtrack_0.54.dpk'...
2025-02-14_07:50:42.42599 [FS] Loading pak '/home/user/www/unvanquished/map-metro-b1-2_0+0.dpk'...
2025-02-14_07:50:42.42655 ^3Warn: [FS] Invalid filename 'sound/metro/metro messaj pickpocket.wav' in pak '/home/user/www/unvanquished/map-metro-b1-2_0+0.dpk'
2025-02-14_07:50:42.42724 [FS] Loading pak '/home/user/www/unvanquished/res-tremulous_1.2.3.dpk'...
2025-02-14_07:50:42.42950 [FS] Loading pak '/home/user/www/unvanquished/res-ambient_0.55.dpk'...
2025-02-14_07:50:42.51454 ^3Warn: CM_SetBorderInward: mixed plane sides
2025-02-14_07:50:42.51460 ^3Warn: CM_SetBorderInward: mixed plane sides
2025-02-14_07:50:42.52013 Loading VM module /home/user/unvanquished/sgame-native-dll.so...
2025-02-14_07:50:42.52293 Loaded sgame VM module in 3 msec
2025-02-14_07:50:42.68055 ^3Warn: SGame VM: Out of memory
2025-02-14_07:50:42.69493 [FS] Loading pak '/home/user/www/unvanquished/mods/deadbeef/mod-deadbeef-light_0.55.0-17.dpk'...
2025-02-14_07:50:42.69525 [FS] Loading pak '/home/user/unvanquished/pkg-release/unvanquished_0.55.1.dpk'...
2025-02-14_07:50:42.69619 [FS] Loading pak '/home/user/unvanquished/pkg-release/tex-common_2.4.dpk'...
2025-02-14_07:50:42.69644 [FS] Loading pak '/home/user/unvanquished/pkg-release/res-players_0.55.dpk'...
2025-02-14_07:50:42.69688 [FS] Loading pak '/home/user/unvanquished/pkg-release/res-legacy_0.55.dpk'...
2025-02-14_07:50:42.69752 [FS] Loading pak '/home/user/unvanquished/pkg-release/res-weapons_0.55.dpk'...
2025-02-14_07:50:42.69839 [FS] Loading pak '/home/user/unvanquished/pkg-release/res-buildables_0.55.dpk'...
2025-02-14_07:50:42.69895 [FS] Loading pak '/home/user/unvanquished/pkg-release/res-voices_0.54.dpk'...
2025-02-14_07:50:42.69924 [FS] Loading pak '/home/user/unvanquished/pkg-release/res-soundtrack_0.54.dpk'...
2025-02-14_07:50:42.69939 ----- Server Shutdown -----