ComfyUI
ComfyUI copied to clipboard
Published
20 hours ago •
comfyanonymous
comfyanonymous
Comfyui killed by oom-killer
Your question
Hi. After last update (5 october) comfyui started killing by oom killer when it's tried load checkpoint. Before update it works fine. workflow.json
Logs
oom-killer logs:
[ 962.892059] pt_main_thread invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=200
[ 962.892067] CPU: 2 UID: 1000 PID: 48024 Comm: pt_main_thread Tainted: P W OE 6.11.2-zen1-1-zen #1 d30fef8841e6fc6d439846897474034e8403afe5
[ 962.892071] Tainted: [P]=PROPRIETARY_MODULE, [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
[ 962.892072] Hardware name: Micro-Star International Co., Ltd. MS-7E27/PRO B650M-P (MS-7E27), BIOS 1.50 12/15/2023
[ 962.892073] Call Trace:
[ 962.892075] <TASK>
[ 962.892078] dump_stack_lvl+0x5d/0x80
[ 962.892082] dump_header+0x44/0x18d
[ 962.892084] oom_kill_process.cold+0x8/0x83
[ 962.892086] out_of_memory+0x29f/0x5b0
[ 962.892089] __alloc_pages_noprof+0x9e3/0xf00
[ 962.892094] alloc_pages_mpol_noprof+0x1c5/0x380
[ 962.892097] folio_alloc_noprof+0x5b/0xa0
[ 962.892098] __filemap_get_folio+0x16f/0x5a0
[ 962.892100] ? srso_alias_return_thunk+0x5/0xfbef5
[ 962.892102] ? page_cache_ra_unbounded+0x10f/0x180
[ 962.892104] filemap_fault+0x317/0x1280
[ 962.892106] ? filemap_map_pages+0x4fa/0x5c0
[ 962.892108] ? srso_alias_return_thunk+0x5/0xfbef5
[ 962.892109] ? __mod_memcg_lruvec_state+0xa0/0x150
[ 962.892112] __do_fault+0x32/0x120
[ 962.892115] do_read_fault+0x44/0x1b0
[ 962.892117] handle_mm_fault+0x1190/0x1500
[ 962.892119] ? mt_find+0x1f0/0x4d0
[ 962.892122] do_user_addr_fault+0x219/0x860
[ 962.892125] exc_page_fault+0x81/0x190
[ 962.892127] asm_exc_page_fault+0x26/0x30
[ 962.892129] RIP: 0033:0x7f1a1aa2bb48
[ 962.892143] Code: 1f 00 48 83 3a 02 75 8b 4d 85 c0 7e 8b 48 8d 79 e0 4c 8d 49 e1 45 31 ff 48 83 e7 e0 4c 8d 6f 20 31 ff 4d 85 c9 7e 25 0f 1f 00 <c5> fe 6f 44 7e 20 c5 fe 6f 0c 7e c5 fe 7f 44 78 20 c5 fe 7f 0c 78
[ 962.892144] RSP: 002b:00007f18ebdfe860 EFLAGS: 00010293
[ 962.892146] RAX: 00007f1570fa0180 RBX: 0000000000000002 RCX: 0000000000215556
[ 962.892147] RDX: 00007f18ebdfe940 RSI: 00007f13bd523f90 RDI: 0000000000054020
[ 962.892148] RBP: 00007f18ebdfe8a0 R08: 0000000000000001 R09: 0000000000215537
[ 962.892149] R10: 0000000000000001 R11: 0000000000000001 R12: 00007f18ebdfe940
[ 962.892150] R13: 0000000000215540 R14: 00007f18ebdfeb90 R15: 0000000000000000
[ 962.892152] </TASK>
[ 962.892153] Mem-Info:
[ 962.892154] active_anon:266715 inactive_anon:1875740 isolated_anon:0
active_file:55606 inactive_file:774367 isolated_file:0
unevictable:0 dirty:472 writeback:0
slab_reclaimable:33355 slab_unreclaimable:66127
mapped:865019 shmem:9877 pagetables:21127
sec_pagetables:1440 bounce:0
kernel_misc_reclaimable:0
free:16120 free_pcp:1601 free_cma:0
[ 962.892158] Node 0 active_anon:1066860kB inactive_anon:7502960kB active_file:222424kB inactive_file:3097468kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:3460076kB dirty:1888kB writeback:0kB shmem:39508kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:962560kB writeback_tmp:0kB kernel_stack:19488kB pagetables:84508kB sec_pagetables:5760kB all_unreclaimable? no
[ 962.892161] Node 0 DMA free:64kB boost:0kB min:64kB low:80kB high:96kB reserved_highatomic:0KB active_anon:548kB inactive_anon:4792kB active_file:1636kB inactive_file:4816kB unevictable:0kB writepending:0kB present:15996kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 962.892164] lowmem_reserve[]: 0 2584 15026 0 0
[ 962.892167] Node 0 DMA32 free:10888kB boost:0kB min:11608kB low:14508kB high:17408kB reserved_highatomic:0KB active_anon:35980kB inactive_anon:1082504kB active_file:15416kB inactive_file:730860kB unevictable:0kB writepending:0kB present:2836932kB managed:2770824kB mlocked:0kB bounce:0kB free_pcp:952kB local_pcp:0kB free_cma:0kB
[ 962.892170] lowmem_reserve[]: 0 0 12442 0 0
[ 962.892172] Node 0 Normal free:53528kB boost:0kB min:55904kB low:69880kB high:83856kB reserved_highatomic:0KB active_anon:1030496kB inactive_anon:6415664kB active_file:204956kB inactive_file:2362516kB unevictable:0kB writepending:1888kB present:13073408kB managed:12747912kB mlocked:0kB bounce:0kB free_pcp:5452kB local_pcp:0kB free_cma:0kB
[ 962.892175] lowmem_reserve[]: 0 0 0 0 0
[ 962.892177] Node 0 DMA: 3*4kB (UM) 3*8kB (U) 0*16kB 1*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 68kB
[ 962.892185] Node 0 DMA32: 1*4kB (M) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (M) 1*2048kB (M) 2*4096kB (M) = 11268kB
[ 962.892192] Node 0 Normal: 1244*4kB (U) 149*8kB (UM) 456*16kB (U) 778*32kB (UM) 3*64kB (UM) 2*128kB (U) 2*256kB (U) 2*512kB (U) 3*1024kB (UM) 5*2048kB (M) 0*4096kB = 53656kB
[ 962.892203] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 962.892204] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 962.892205] 840633 total pagecache pages
[ 962.892206] 775 pages in swap cache
[ 962.892206] Free swap = 44215544kB
[ 962.892207] Total swap = 49088504kB
[ 962.892207] 3981584 pages RAM
[ 962.892208] 0 pages HighMem/MovableOnly
[ 962.892208] 98060 pages reserved
[ 962.892209] 0 pages cma reserved
[ 962.892209] 0 pages hwpoisoned
[ 962.892210] Tasks state (memory values in pages):
[ 962.892210] [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
[ 962.892215] [ 511] 0 511 29781 423 32 391 0 114688 288 -250 systemd-journal
[ 962.892218] [ 539] 0 539 3572 393 32 361 0 77824 160 0 systemd-userdbd
[ 962.892220] [ 559] 0 559 9029 427 0 427 0 90112 1088 -1000 systemd-udevd
[ 962.892223] [ 851] 978 851 3711 427 64 363 0 69632 128 -900 systemd-oomd
[ 962.892225] [ 852] 975 852 22264 347 0 347 0 81920 256 0 systemd-timesyn
[ 962.892227] [ 862] 81 862 1866 332 0 332 0 53248 256 -900 dbus-broker-lau
[ 962.892229] [ 863] 81 863 1620 290 10 280 0 49152 608 -900 dbus-broker
[ 962.892230] [ 864] 0 864 42638 891 32 859 0 94208 640 -999 ananicy-cpp
[ 962.892232] [ 865] 0 865 304180 1501 192 1309 0 249856 800 0 NetworkManager
[ 962.892234] [ 867] 974 867 1556 307 0 307 0 57344 96 0 avahi-daemon
[ 962.892236] [ 870] 0 870 20209 365 0 365 0 65536 96 0 irqbalance
[ 962.892237] [ 873] 102 873 78065 1130 0 1130 0 118784 992 0 polkitd
[ 962.892239] [ 874] 0 874 5408 2202 1781 421 0 86016 2208 0 preload
[ 962.892241] [ 876] 0 876 3787 349 17 332 0 77824 256 0 systemd-logind
[ 962.892242] [ 877] 0 877 3715 352 0 352 0 73728 192 0 systemd-machine
[ 962.892244] [ 878] 0 878 17679 554 0 554 0 126976 448 -900 virtlogd
[ 962.892246] [ 884] 974 884 1556 180 1 179 0 57344 64 0 avahi-daemon
[ 962.892247] [ 953] 0 953 97284 531 0 531 0 135168 416 0 ModemManager
[ 962.892249] [ 3624] 0 3624 3948 361 0 361 0 77824 384 0 wpa_supplicant
[ 962.892251] [ 3632] 0 3632 318906 124 0 124 0 192512 5375 0 torrserver
[ 962.892252] [ 3638] 0 3638 38351 849 128 721 0 143360 544 0 sddm
[ 962.892254] [ 3672] 0 3672 6589471 16679 7790 7315 1574 614400 9088 0 Xorg
[ 962.892256] [ 4203] 0 4203 19182 976 96 880 0 139264 576 0 nmbd
[ 962.892257] [ 6481] 0 6481 19356 1222 64 1158 0 135168 512 0 sddm-helper
[ 962.892259] [ 6487] 1000 6487 5074 383 64 319 0 81920 704 100 systemd
[ 962.892261] [ 6489] 1000 6489 5178 297 56 241 0 73728 352 100 (sd-pam)
[ 962.892263] [ 6497] 1000 6497 230889 1318 96 1222 0 266240 768 200 appimagelaunche
[ 962.892264] [ 6498] 1000 6498 3093 326 32 294 0 53248 32 200 gamemoded
[ 962.892266] [ 6506] 1000 6506 1735 336 32 304 0 61440 32 200 dbus-broker-lau
[ 962.892268] [ 6514] 1000 6514 1521 325 82 243 0 53248 448 200 dbus-broker
[ 962.892269] [ 6544] 1000 6544 102442 3459 159 3300 0 331776 4864 0 kwalletd6
[ 962.892271] [ 6546] 1000 6546 54487 1569 128 1441 0 212992 1088 0 startplasma-x11
[ 962.892278] [ 6629] 1000 6629 86608 3738 69 3669 0 253952 3520 200 kglobalacceld
[ 962.892280] [ 6643] 1000 6643 67168220 1919 13 1906 0 290816 2496 200 baloo_file
[ 962.892282] [ 6653] 1000 6653 173511 1153 32 1121 0 204800 736 200 xdg-desktop-por
[ 962.892284] [ 6687] 1000 6687 151147 1076 32 1044 0 163840 128 200 xdg-document-po
[ 962.892286] [ 6703] 0 6703 116907 564 0 564 0 159744 672 0 udisksd
[ 962.892288] [ 6704] 1000 6704 76467 1024 0 1024 0 114688 96 200 xdg-permission-
[ 962.892290] [ 6716] 1000 6716 640 266 0 266 0 49152 0 200 fusermount3
[ 962.892292] [ 6723] 1000 6723 132872 1349 71 1278 0 241664 3520 200 xdg-desktop-por
[ 962.892294] [ 6804] 0 6804 79154 1082 0 1082 0 135168 704 0 upowerd
[ 962.892295] [ 6818] 1000 6818 80570 972 0 972 0 147456 384 200 gvfsd
[ 962.892297] [ 6821] 1000 6821 98640 3517 100 3417 0 294912 4160 200 ksmserver
[ 962.892299] [ 6825] 1000 6825 439505 3948 262 3686 0 503808 7136 200 kded6
[ 962.892300] [ 6827] 1000 6827 375488 27595 3083 24512 0 1130496 31293 200 kwin_x11
[ 962.892302] [ 6856] 1000 6856 132796 1014 0 1014 0 155648 192 200 gvfsd-fuse
[ 962.892304] [ 6871] 1000 6871 94792 578 0 578 0 118784 160 200 at-spi-bus-laun
[ 962.892306] [ 6877] 1000 6877 1703 321 0 321 0 53248 64 200 dbus-broker-lau
[ 962.892307] [ 6878] 1000 6878 1054 269 0 269 0 49152 64 200 dbus-broker
[ 962.892309] [ 6879] 1000 6879 58094 655 64 591 0 98304 96 200 at-spi2-registr
[ 962.892311] [ 6890] 133 6890 21513 353 32 321 0 69632 32 0 rtkit-daemon
[ 962.892312] [ 6943] 1000 6943 1086685 35886 2832 33054 0 2265088 100928 200 plasmashell
[ 962.892314] [ 6950] 1000 6950 57271 561 32 529 0 90112 96 200 dconf-service
[ 962.892316] [ 6986] 1000 6986 9704 1116 66 1026 24 98304 1312 200 pipewire
[ 962.892318] [ 6988] 1000 6988 171940 1318 0 1318 0 303104 1952 200 wireplumber
[ 962.892320] [ 6989] 1000 6989 26115 1107 62 1045 0 106496 2592 200 pipewire-pulse
[ 962.892321] [ 7003] 1000 7003 1953 394 0 394 0 61440 64 200 xsettingsd
[ 962.892323] [ 7004] 1000 7004 55572 2342 32 2310 0 147456 608 200 kscreen_backend
[ 962.892324] [ 7007] 1000 7007 167235 3395 26 3369 0 315392 3936 200 kactivitymanage
[ 962.892326] [ 7008] 1000 7008 55971 2354 0 2354 0 147456 608 200 gmenudbusmenupr
[ 962.892328] [ 7009] 1000 7009 254365 7749 98 7651 0 487424 8736 200 polkit-kde-auth
[ 962.892330] [ 7010] 1000 7010 227702 3428 0 3428 0 385024 4768 200 org_kde_powerde
[ 962.892331] [ 7011] 1000 7011 280315 3468 11 3457 0 450560 10624 200 xdg-desktop-por
[ 962.892333] [ 7012] 1000 7012 55500 2396 0 2396 0 143360 512 200 xembedsniproxy
[ 962.892335] [ 7118] 1000 7118 146113 1534 175 1359 0 315392 7232 200 firewall-applet
[ 962.892336] [ 7119] 1000 7119 263348 3443 1 3442 0 421888 4928 200 garuda-system-m
[ 962.892339] [ 7129] 1000 7129 101892 3466 156 3310 0 327680 4576 200 kdeconnectd
[ 962.892340] [ 7159] 1000 7159 77462 1070 0 1070 0 118784 160 200 agent
[ 962.892342] [ 7160] 1000 7160 100875 3266 13 3253 0 323584 4352 200 kaccess
[ 962.892344] [ 7161] 1000 7161 100943 3295 0 3295 0 319488 4384 200 DiscoverNotifie
[ 962.892346] [ 7191] 952 7191 78738 1129 0 1129 0 126976 1088 0 colord
[ 962.892348] [ 7317] 0 7317 22253 998 32 966 0 155648 800 0 smbd
[ 962.892349] [ 7322] 0 7322 21627 830 38 792 0 126976 768 0 smbd-notifyd
[ 962.892351] [ 7323] 0 7323 21625 835 38 797 0 122880 768 0 smbd-cleanupd
[ 962.892353] [ 7324] 0 7324 76910 1053 0 1053 0 114688 192 0 power-profiles-
[ 962.892355] [ 10760] 1000 10760 1455 735 0 735 0 49152 0 200 cat
[ 962.892357] [ 10761] 1000 10761 1455 738 0 738 0 53248 0 200 cat
[ 962.892359] [ 11428] 1000 11428 1455 742 0 742 0 53248 0 200 cat
[ 962.892360] [ 11429] 1000 11429 1455 717 0 717 0 53248 0 200 cat
[ 962.892362] [ 11930] 0 11930 179054 1254 126 1128 0 319488 13056 0 pamac-daemon
[ 962.892364] [ 15553] 1000 15553 58242 1032 0 1032 0 98304 128 200 gvfsd-metadata
[ 962.892365] [ 15586] 1000 15586 67210032 3954 145 3809 0 413696 4416 200 baloorunner
[ 962.892367] [ 18557] 1000 18557 318408 5637 211 5329 97 491520 6784 200 dolphin
[ 962.892369] [ 21902] 1000 21902 8696727 42364 11121 31145 98 1998848 30338 200 yandex_browser
[ 962.892371] [ 21907] 1000 21907 1455 720 0 720 0 49152 0 200 cat
[ 962.892373] [ 21908] 1000 21908 1455 701 0 701 0 49152 0 200 cat
[ 962.892375] [ 21910] 1000 21910 8394196 355 32 323 0 73728 32 200 chrome_crashpad
[ 962.892376] [ 21912] 1000 21912 8391887 352 32 320 0 65536 32 200 chrome_crashpad
[ 962.892378] [ 21918] 1000 21918 8603976 1544 242 1302 0 626688 4512 200 yandex_browser
[ 962.892380] [ 21919] 1000 21919 8604802 1480 179 1301 0 638976 4544 200 yandex_browser
[ 962.892381] [ 21921] 1000 21921 8604841 507 146 361 0 303104 4608 200 yandex_browser
[ 962.892383] [ 21951] 1000 21951 8744787 26564 5635 17394 3535 1835008 31295 200 yandex_browser
[ 962.892385] [ 21953] 1000 21953 8553074 9915 2230 7609 76 905216 5885 200 yandex_browser
[ 962.892387] [ 21963] 1000 21963 8617811 2302 375 1927 0 598016 5152 200 yandex_browser
[ 962.892388] [ 21978] 1000 21978 297329570 4556 854 3637 65 950272 8604 300 yandex_browser
[ 962.892390] [ 22027] 1000 22027 296855304 4235 614 3516 105 970752 7856 300 yandex_browser
[ 962.892392] [ 22043] 1000 22043 8636529 3897 211 3686 0 462848 7328 200 yandex_browser
[ 962.892394] [ 22050] 1000 22050 296850353 4204 568 3619 17 937984 8452 300 yandex_browser
[ 962.892396] [ 22061] 1000 22061 296853188 4027 692 3271 64 946176 7752 300 yandex_browser
[ 962.892398] [ 22071] 1000 22071 294731187 2191 252 1939 0 589824 5088 200 yandex_browser
[ 962.892399] [ 22090] 1000 22090 297333232 4826 1035 3707 84 1134592 21432 300 yandex_browser
[ 962.892401] [ 22195] 1000 22195 8555875 1942 256 1686 0 729088 4224 200 yandex_browser
[ 962.892403] [ 24455] 1000 24455 144880 1132 0 1132 0 196608 608 200 gvfs-udisks2-vo
[ 962.892404] [ 24463] 1000 24463 135964 1040 0 1040 0 188416 448 200 gvfsd-trash
[ 962.892406] [ 24479] 1000 24479 117593 992 0 992 0 176128 480 200 gvfsd-network
[ 962.892408] [ 28512] 1000 28512 8615314 2556 314 2242 0 663552 16978 200 yandex_browser
[ 962.892410] [ 29913] 1000 29913 297328307 6608 1428 5139 41 1089536 19262 300 yandex_browser
[ 962.892412] [ 30023] 1000 30023 296854899 19257 5334 13906 17 1028096 7901 300 yandex_browser
[ 962.892413] [ 30035] 1000 30035 296852575 4611 472 4069 70 917504 6540 300 yandex_browser
[ 962.892415] [ 31152] 1000 31152 296855094 4774 560 4214 0 991232 9355 300 yandex_browser
[ 962.892417] [ 31153] 1000 31153 296856994 4914 728 4152 34 999424 8270 300 yandex_browser
[ 962.892418] [ 31159] 1000 31159 296855194 4730 657 4073 0 950272 7732 300 yandex_browser
[ 962.892420] [ 31218] 1000 31218 296862296 10265 3433 6790 42 1118208 11849 300 yandex_browser
[ 962.892422] [ 31248] 1000 31248 8615314 2514 326 2174 14 507904 4625 200 yandex_browser
[ 962.892424] [ 31254] 1000 31254 297343064 6191 1339 4721 131 1413120 35517 300 yandex_browser
[ 962.892425] [ 31282] 1000 31282 296850353 4197 319 3871 7 909312 6704 300 yandex_browser
[ 962.892427] [ 31298] 1000 31298 296858908 5384 1021 4307 56 1224704 18687 300 yandex_browser
[ 962.892429] [ 41042] 1000 41042 296859026 7564 2154 5248 162 1277952 21300 300 yandex_browser
[ 962.892431] [ 41079] 1000 41079 296852662 4785 362 4328 95 933888 7668 300 yandex_browser
[ 962.892432] [ 41096] 1000 41096 296867426 33625 16264 13624 3737 1273856 6989 300 yandex_browser
[ 962.892434] [ 44345] 1000 44345 241037 13900 843 11553 1504 499712 7104 200 konsole
[ 962.892436] [ 44354] 1000 44354 58713 875 0 875 0 110592 576 200 fish
[ 962.892438] [ 47963] 1000 47963 13474872 2816691 2059588 756495 608 33275904 623072 200 pt_main_thread
[ 962.892440] [ 48404] 1000 48404 296835945 3694 356 3293 45 745472 5780 300 yandex_browser
[ 962.892441] [ 48416] 0 48416 3708 462 64 398 0 69632 128 0 systemd-userwor
[ 962.892443] [ 48417] 0 48417 3708 462 64 398 0 73728 128 0 systemd-userwor
[ 962.892445] [ 48418] 0 48418 3673 526 96 430 0 69632 96 0 systemd-userwor
[ 962.892446] [ 48714] 1000 48714 297313130 4513 532 3896 85 811008 6132 300 yandex_browser
[ 962.892448] [ 48739] 1000 48739 297313916 4964 632 4230 102 856064 6380 300 yandex_browser
[ 962.892450] [ 48740] 1000 48740 297318014 4688 544 4143 1 888832 7760 300 yandex_browser
[ 962.892451] [ 48763] 1000 48763 297313130 4564 518 3993 53 851968 6669 300 yandex_browser
[ 962.892453] [ 49079] 1000 49079 297313130 5538 418 5088 32 823296 6344 300 yandex_browser
[ 962.892455] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/[email protected]/app.slice/[email protected],task=pt_main_thread,pid=47963,uid=1000
[ 962.892527] Out of memory: Killed process 47963 (pt_main_thread) total-vm:53899488kB, anon-rss:8236816kB, file-rss:3025980kB, shmem-rss:2432kB, UID:1000 pgtables:32496kB oom_score_adj:200
### Other
My total RAM: 16GB + 32Gb swap file + 14,8Gb zram.
