ert
ert copied to clipboard
Detect OOM realization kill
When a realization is killed due to out-of-memory, this stays unrecognized and we should provide a proper treatment when OOM appears.
The OOM-kill can be reproduced on a local Linux workstation with:
[havb@be-lx139213:~]$ perl -wE 'my @xs; for (1..2**20) { push @xs, q{a} x 2**20 }; say scalar @xs;'
Killed
(thanks internet, just doing it simple in Python will just give a MemoryError)
After the incident, you can do
[havb@be-lx139213:~]$ dmesg --time-format iso | grep oom
2024-04-29T14:55:41,678628+02:00 pool invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
2024-04-29T14:55:41,678661+02:00 oom_kill_process.cold.33+0xb/0x10
2024-04-29T14:55:41,678996+02:00 [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
2024-04-29T14:55:41,679685+02:00 oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-21417.slice/[email protected],task=perl,pid=1440763,uid=21417
2024-04-29T14:55:41,679700+02:00 Out of memory: Killed process 1440763 (perl) total-vm:87033376kB, anon-rss:60356340kB, file-rss:0kB, shmem-rss:0kB, UID:21417 pgtables:169956kB oom_score_adj:0
2024-04-29T14:55:46,947445+02:00 oom_reaper: reaped process 1440763 (perl), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
(the iso-format of dmesg is RHEL8 only)