ert icon indicating copy to clipboard operation
ert copied to clipboard

Detect OOM realization kill

Open xjules opened this issue 1 year ago • 1 comments

When a realization is killed due to out-of-memory, this stays unrecognized and we should provide a proper treatment when OOM appears.

xjules avatar Apr 29 '24 10:04 xjules

The OOM-kill can be reproduced on a local Linux workstation with:

[havb@be-lx139213:~]$ perl -wE 'my @xs; for (1..2**20) { push @xs, q{a} x 2**20 }; say scalar @xs;'
Killed

(thanks internet, just doing it simple in Python will just give a MemoryError)

After the incident, you can do

[havb@be-lx139213:~]$ dmesg --time-format iso  | grep oom
2024-04-29T14:55:41,678628+02:00 pool invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
2024-04-29T14:55:41,678661+02:00  oom_kill_process.cold.33+0xb/0x10
2024-04-29T14:55:41,678996+02:00 [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
2024-04-29T14:55:41,679685+02:00 oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-21417.slice/[email protected],task=perl,pid=1440763,uid=21417
2024-04-29T14:55:41,679700+02:00 Out of memory: Killed process 1440763 (perl) total-vm:87033376kB, anon-rss:60356340kB, file-rss:0kB, shmem-rss:0kB, UID:21417 pgtables:169956kB oom_score_adj:0
2024-04-29T14:55:46,947445+02:00 oom_reaper: reaped process 1440763 (perl), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

(the iso-format of dmesg is RHEL8 only)

berland avatar Apr 29 '24 13:04 berland