rendertron
rendertron copied to clipboard
Published
20 hours ago •
GoogleChrome
GoogleChrome
Server not responding after running 1 or 2 days, suspect memory leak
I setup a ubuntu server which have 2GB ram and 2GB swap, after i setup the rendertron server and let it run like 1 or 2 days, it just suddenly dies and cannot go to the rendertron website, cannot ssh into it and cannot ping it also, all not responding.
After restart server and i check the syslog:
Aug 1 02:52:38 ip-172-26-14-5 fwupd[209453]: 02:51:47:0174 GLib-GIO Error releasing name org.freedesktop.fwupd: Timeout was reached Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241168] chrome invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241172] CPU: 0 PID: 114853 Comm: chrome Not tainted 5.8.0-1038-aws #40~20.04.1-Ubuntu Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241173] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006 Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241247] Mem-Info: Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241250] active_anon:331704 inactive_anon:112434 isolated_anon:0 Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241250] active_file:137 inactive_file:325 isolated_file:0 Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241250] unevictable:5766 dirty:0 writeback:0 Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241250] slab_reclaimable:10443 slab_unreclaimable:18936 Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241250] mapped:2131 shmem:3 pagetables:5968 bounce:0 Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241250] free:13206 free_pcp:246 free_cma:0 Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241253] Node 0 active_anon:1326816kB inactive_anon:449736kB active_file:548kB inactive_file:1300kB unevictable:23064kB isolated(anon):0kB is> Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241253] Node 0 DMA free:8176kB min:352kB low:440kB high:528kB reserved_highatomic:0KB active_anon:6212kB inactive_anon:68kB active_file:0kB > Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241256] lowmem_reserve[]: 0 1957 1957 1957 1957 Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241257] Node 0 DMA32 free:44648kB min:44700kB low:55872kB high:67044kB reserved_highatomic:2048KB active_anon:1320604kB inactive_anon:449668> Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241260] lowmem_reserve[]: 0 0 0 0 0 Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241261] Node 0 DMA: 8*4kB (UME) 6*8kB (UME) 4*16kB (UME) 1*32kB (U) 3*64kB (ME) 3*128kB (UE) 1*256kB (M) 2*512kB (ME) 2*1024kB (ME) 2*2048kB> Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241267] Node 0 DMA32: 574*4kB (UMEH) 306*8kB (UME) 250*16kB (UME) 210*32kB (UME) 64*64kB (UME) 196*128kB (UME) 0*256kB 0*512kB 0*1024kB 0*20> Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241273] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241273] 3701 total pagecache pages Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241275] 41 pages in swap cache Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241276] Swap cache stats: add 50688876, delete 50688835, find 14561146/26816956 Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241276] Free swap = 0kB Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241276] Total swap = 2097148kB Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241277] 524189 pages RAM Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241277] 0 pages HighMem/MovableOnly Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241278] 16233 pages reserved Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241278] 0 pages hwpoisoned Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241279] Tasks state (memory values in pages): Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241279] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241282] [ 273] 0 273 70052 4499 94208 0 -1000 multipathd Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241284] [ 419] 0 419 59331 164 90112 229 0 accounts-daemon Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241286] [ 420] 0 420 637 168 45056 35 0 acpid Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241287] [ 424] 0 424 2136 410 53248 53 0 cron Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241288] [ 428] 103 428 1934 687 57344 119 -900 dbus-daemon Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241290] [ 438] 0 438 7321 795 90112 1928 0 networkd-dispat Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241291] [ 440] 104 440 56127 132 86016 269 0 rsyslogd Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241292] [ 444] 0 444 4176 650 73728 202 0 systemd-logind Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241293] [ 450] 0 450 950 542 45056 49 0 atd Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241295] [ 465] 0 465 1840 384 53248 36 0 agetty Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241296] [ 473] 0 473 1459 389 45056 33 0 agetty Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241297] [ 496] 0 496 58181 167 81920 114 0 polkitd Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241298] [ 527] 0 527 27029 825 106496 1945 0 unattended-upgr Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241300] [ 608] 0 608 3046 566 61440 235 -1000 sshd Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241301] [ 2666] 0 2666 92492 722 208896 9905 0 glances Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241302] [ 114834] 0 114834 148628 560 790528 4698 0 node Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241303] [ 114842] 0 114842 218031 2009 1617920 22126 0 node Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241305] [ 114853] 0 114853 4377556 50948 2023424 32796 0 chrome Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241306] [ 114855] 0 114855 4259551 722 372736 2707 0 chrome Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241307] [ 114856] 0 114856 4259551 557 376832 2691 0 chrome Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241308] [ 114875] 0 114875 5358670 12493 2646016 70575 300 chrome Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241309] [ 169992] 0 169992 5358051 17358 2629632 64091 300 chrome Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241310] [ 170050] 0 170050 5353154 15738 2428928 57580 300 chrome Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241311] [ 170182] 0 170182 5362892 13785 2486272 57936 300 chrome Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241312] [ 170614] 0 170614 5374424 18332 3088384 78968 300 chrome Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241313] [ 170625] 0 170625 5354631 13583 2387968 62330 300 chrome Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241314] [ 170671] 0 170671 5371479 23359 2998272 67712 300 chrome Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241316] [ 173581] 100 173581 6653 737 73728 228 0 systemd-network Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241317] [ 173592] 0 173592 24685 465 221184 331 -250 systemd-journal Aug 1 03:17:00 ip-172-26-14-5 kernel: [1127181.241319] [ 181106] 0 181106 4669 457 57344 252 -1000 systemd-udevd Aug 1 03:17:01 ip-172-26-14-5 kernel: [1127181.241320] [ 199840] 101 199840 5976 729 86016 968 0 systemd-resolve Aug 1 03:17:01 ip-172-26-14-5 kernel: [1127181.241321] [ 208054] 102 208054 22559 480 81920 165 0 systemd-timesyn Aug 1 03:17:01 ip-172-26-14-5 kernel: [1127181.241322] [ 209402] 0 209402 4279851 275290 3076096 2988 200 chrome Aug 1 03:17:01 ip-172-26-14-5 kernel: [1127181.241347] [ 209453] 0 209453 93487 672 172032 139 0 fwupd Aug 1 03:17:01 ip-172-26-14-5 kernel: [1127181.241348] [ 209504] 0 209504 2521 512 53248 43 0 cron Aug 1 03:17:01 ip-172-26-14-5 kernel: [1127181.241349] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/rendertron.service,t> Aug 1 03:17:01 ip-172-26-14-5 kernel: [1127181.241363] Out of memory: Killed process 209402 (chrome) total-vm:17119404kB, anon-rss:1100276kB, file-rss:884kB, shmem-rss:0kB, UID:0 pgtables> Aug 1 03:17:01 ip-172-26-14-5 kernel: [1127181.584261] oom_reaper: reaped process 209402 (chrome), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
It appear that oom is kick in and killing chrome process, but it still hang becoz it is not responding until i restart the server.
This is the config.json i set for rendertron (rendertron commit id: 7999fc5) :
{ "timeout":30000, "port":80, "closeBrowser":false, "cache": "filesystem", "cacheConfig": { "cacheDurationMinutes": 120, "cacheMaxEntries": 1000 } }
@nogamenofun98 do you mind testing it the changes from PR#813 to see if it any better/worse?
@gravi2 hi, sorry for the late testing, i got pull the PR#813 and run "npm i" and "npm run build" d, so far seem ok. I go update due to the rendertron service can access to the home page but when click render as screenshot or render as page, it will stuck. After update now is back to ok
In general, as we're deprecating the project, you should look into alternative approaches to rendering on the web.
