gpdb
gpdb copied to clipboard
Published
20 hours ago •
greenplum-db
greenplum-db
The global deadlock detected process of the greenplum master node has abnormal RSS memory usage.
1、02:09:43, See the deadlock information from the log
$ cat gpdb-2024-04-18_000000.csv | grep deadlock
2024-04-18 02:09:43.246943 UTC,,,p26809,th-1525422016,,,,0,con380,cmd5036,seg-1,,,,sx1,"LOG","00000","global deadlock detected! Final graph is :{""seg0"": [""p1241 of dtx3430845286 con1088326 waits for a transactionid lock on ShareLock mode, blocked by p1232 of dtx3430845283 con1088326"",""p1232 of dtx3430845283 con1088324 waits for a transactionid lock on ShareLock mode, blocked by p1225 of dtx3430845280 con1088324"",""p1237 of dtx3430845285 con1088325 waits for a transactionid lock on ShareLock mode, blocked by p1222 of dtx3430845278 con1088325"",""p1223 of dtx3430845279 con1088321 waits for a transactionid lock on ShareLock mode, blocked by p1232 of dtx3430845283 con1088321"",""p1250 of dtx3430845289 con1088328 waits for a transactionid lock on ShareLock mode, blocked by p1237 of dtx3430845285 con1088328""],""seg1"": [""p8451 of dtx3430845279 con1088321 waits for a transactionid lock on ShareLock mode, blocked by p8466 of dtx3430845285 con1088321"",""p8466 of dtx3430845285 con1088325 waits for a transactionid lock on ShareLock mode, blocked by p8478 of dtx3430845289 con1088325"",""p8460 of dtx3430845283 con1088324 waits for a transactionid lock on ShareLock mode, blocked by p8450 of dtx3430845278 con1088324"",""p8478 of dtx3430845289 con1088328 waits for a transactionid lock on ShareLock mode, blocked by p8470 of dtx3430845286 con1088328"",""p8453 of dtx3430845280 con1088322 waits for a transactionid lock on ShareLock mode, blocked by p8478 of dtx3430845289 con1088322"",""p8457 of dtx3430845281 con1088323 waits for a transactionid lock on ShareLock mode, blocked by p8450 of dtx3430845278 con1088323""],""seg2"": [""p26938 of dtx3430845281 con1088323 waits for a transactionid lock on ShareLock mode, blocked by p26933 of dtx3430845279 con1088323"",""p26959 of dtx3430845289 con1088328 waits for a transactionid lock on ShareLock mode, blocked by p26952 of dtx3430845286 con1088328"",""p26952 of dtx3430845286 con1088326 waits for a transactionid lock on ShareLock mode, blocked by p26933 of dtx3430845279 con1088326"",""p26947 of dtx3430845285 con1088325 waits for a transactionid lock on ShareLock mode, blocked by p26938 of dtx3430845281 con1088325"",""p26932 of dtx3430845278 con1088320 waits for a transactionid lock on ShareLock mode, blocked by p26941 of dtx3430845283 con1088320""]}",,,,,,,0,,"gddbackend.c",228,
The log shows that the process id is p26809
2、02:09:50, Abnormal growth of master memory
3、02:17:55, kernel invoked oom-killer
Apr 18 02:17:55 mdw kernel: tuned invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Apr 18 02:17:55 mdw kernel: CPU: 51 PID: 1965 Comm: tuned Tainted: G E 5.4.258-1.el7.elrepo.x86_64 #1
Apr 18 02:17:55 mdw kernel: Hardware name: Amazon EC2 c6i.16xlarge/, BIOS 1.0 10/16/2017
Apr 18 02:17:55 mdw kernel: Call Trace:
Apr 18 02:17:55 mdw kernel: dump_stack+0x6d/0x8b
Apr 18 02:17:55 mdw kernel: dump_header+0x4f/0x1eb
Apr 18 02:17:55 mdw kernel: oom_kill_process.cold+0xb/0x10
Apr 18 02:17:55 mdw kernel: out_of_memory+0x1cf/0x500
Apr 18 02:17:55 mdw kernel: __alloc_pages_slowpath+0xdaa/0xeb0
Apr 18 02:17:55 mdw kernel: __alloc_pages_nodemask+0x2c6/0x300
Apr 18 02:17:55 mdw kernel: alloc_pages_current+0x87/0xe0
Apr 18 02:17:55 mdw kernel: __page_cache_alloc+0x72/0x90
Apr 18 02:17:55 mdw kernel: pagecache_get_page+0xbf/0x300
Apr 18 02:17:55 mdw kernel: filemap_fault+0x6ea/0xac0
Apr 18 02:17:55 mdw kernel: ? unlock_page_memcg+0x12/0x20
Apr 18 02:17:55 mdw kernel: ? alloc_set_pte+0x114/0x5c0
Apr 18 02:17:55 mdw kernel: ? _cond_resched+0x19/0x30
Apr 18 02:17:55 mdw kernel: ? down_read+0x13/0xa0
Apr 18 02:17:55 mdw kernel: __xfs_filemap_fault+0x6c/0x200 [xfs]
Apr 18 02:17:55 mdw kernel: xfs_filemap_fault+0x37/0x40 [xfs]
Apr 18 02:17:55 mdw kernel: __do_fault+0x3c/0x170
Apr 18 02:17:55 mdw kernel: do_fault+0x24b/0x620
Apr 18 02:17:55 mdw kernel: __handle_mm_fault+0x478/0x720
Apr 18 02:17:55 mdw kernel: handle_mm_fault+0xca/0x200
Apr 18 02:17:55 mdw kernel: do_user_addr_fault+0x1e2/0x430
Apr 18 02:17:55 mdw kernel: __do_page_fault+0x42/0x50
Apr 18 02:17:55 mdw kernel: do_page_fault+0x36/0x120
Apr 18 02:17:55 mdw kernel: do_async_page_fault+0x43/0xd0
Apr 18 02:17:55 mdw kernel: ? prepare_exit_to_usermode+0x8c/0xb0
Apr 18 02:17:55 mdw kernel: async_page_fault+0x3e/0x50
Apr 18 02:17:55 mdw kernel: RIP: 0033:0x7f855a6e9770
Apr 18 02:17:55 mdw kernel: Code: Bad RIP value.
Apr 18 02:17:55 mdw kernel: RSP: 002b:00007f8548b05dd8 EFLAGS: 00010202
Apr 18 02:17:55 mdw kernel: RAX: 00007f855a96fb60 RBX: 00007f855aa58800 RCX: 0000000000000000
Apr 18 02:17:55 mdw kernel: RDX: 00007f855a99a1c8 RSI: 0000000000000001 RDI: 00007f85482d6bd0
Apr 18 02:17:55 mdw kernel: RBP: 00007f85482d6bd0 R08: 00007f8548b05dd0 R09: 00007f8548b05b80
Apr 18 02:17:55 mdw kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 00007f85520df360
Apr 18 02:17:55 mdw kernel: R13: 0000000000000001 R14: 00007f853c0131e0 R15: 00007f8552955488
......
Apr 18 02:17:55 mdw kernel: Tasks state (memory values in pages):
Apr 18 02:17:55 mdw kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
......
Apr 18 02:17:55 mdw kernel: [ 26809] 996 26809 33710005 30046023 241467392 0 0 postgres
......
Apr 18 02:17:55 mdw kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=postgres,pid=26809,uid=996
Apr 18 02:17:55 mdw kernel: Out of memory: Killed process 26809 (postgres) total-vm:134840020kB, anon-rss:120147564kB, file-rss:0kB, shmem-rss:36528kB, UID:996 pgtables:235808kB oom_score_adj:0
Apr 18 02:17:55 mdw systemd: Started Push metric to node_exporter textfiles.
Apr 18 02:17:59 mdw kernel: oom_reaper: reaped process 26809 (postgres), now anon-rss:0kB, file-rss:0kB, shmem-rss:36528kB
26809 The process memory usage is abnormal and is killed by oom.
anon-rss:120147564kB = 114GB
4、02:18:00, Memory usage starts to drop
ENV
postgres=# select version();
version
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
------
PostgreSQL 12.12 (Greenplum Database 7.0.0 build commit:0a7a3566873325aca1789ae6f818c80f17a9402d) on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44), 64-bit compiled on Oct 19 2023 16:40:25 Bhuvne
sh C.
(1 row)
$ pg_config
BINDIR = /usr/local/greenplum-db/bin
DOCDIR = /usr/local/greenplum-db/share/doc/postgresql
HTMLDIR = /usr/local/greenplum-db/share/doc/postgresql
INCLUDEDIR = /usr/local/greenplum-db/include
PKGINCLUDEDIR = /usr/local/greenplum-db/include/postgresql
INCLUDEDIR-SERVER = /usr/local/greenplum-db/include/postgresql/server
LIBDIR = /usr/local/greenplum-db/lib
PKGLIBDIR = /usr/local/greenplum-db/lib/postgresql
LOCALEDIR = /usr/local/greenplum-db/share/locale
MANDIR = /usr/local/greenplum-db/share/man
SHAREDIR = /usr/local/greenplum-db/share/postgresql
SYSCONFDIR = /usr/local/greenplum-db/etc/postgresql
PGXS = /usr/local/greenplum-db/lib/postgresql/pgxs/src/makefiles/pgxs.mk
CONFIGURE = '--prefix=/usr/local/gpdb' '--disable-orca' '--disable-gpcloud' '--disable-gpfdist' '--with-perl' '--with-python' '--with-libxml' 'CFLAGS=-g -O3 -march=native' 'CXXFLAGS=-g -O3 -march=native'
CC = gcc -std=gnu99
CPPFLAGS = -D_GNU_SOURCE -I/usr/include/libxml2
CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-unused-but-set-variable -g -O3 -march=native -Werror=uninitialized -Werror=implicit-function-declaration
CFLAGS_SL = -fPIC
LDFLAGS = -Wl,--as-needed -Wl,-rpath,'/usr/local/gpdb/lib',--enable-new-dtags
LDFLAGS_EX =
LDFLAGS_SL =
LIBS = -lpgcommon -lpgport -lpthread -lbz2 -lxml2 -lrt -lz -lreadline -lrt -lcrypt -ldl -lm -lcurl -lzstd
VERSION = PostgreSQL 12.12
But so far I haven’t found the steps to reproduce the problem. Does need any other information?
[postgres@mdw log]$ cat gpdb-2024-04-18_000000.csv | grep 'deadlock'
2024-04-18 02:09:43.246943 UTC,,,p26809,th-1525422016,,,,0,con380,cmd5036,seg-1,,,,sx1,"LOG","00000","global deadlock detected! Final graph is :{""seg0"": [""p1241 of dtx3430845286 con1088326 waits for a transactionid lock on ShareLock mode, blocked by p1232 of dtx3430845283 con1088326"",""p1232 of dtx3430845283 con1088324 waits for a transactionid lock on ShareLock mode, blocked by p1225 of dtx3430845280 con1088324"",""p1237 of dtx3430845285 con1088325 waits for a transactionid lock on ShareLock mode, blocked by p1222 of dtx3430845278 con1088325"",""p1223 of dtx3430845279 con1088321 waits for a transactionid lock on ShareLock mode, blocked by p1232 of dtx3430845283 con1088321"",""p1250 of dtx3430845289 con1088328 waits for a transactionid lock on ShareLock mode, blocked by p1237 of dtx3430845285 con1088328""],""seg1"": [""p8451 of dtx3430845279 con1088321 waits for a transactionid lock on ShareLock mode, blocked by p8466 of dtx3430845285 con1088321"",""p8466 of dtx3430845285 con1088325 waits for a transactionid lock on ShareLock mode, blocked by p8478 of dtx3430845289 con1088325"",""p8460 of dtx3430845283 con1088324 waits for a transactionid lock on ShareLock mode, blocked by p8450 of dtx3430845278 con1088324"",""p8478 of dtx3430845289 con1088328 waits for a transactionid lock on ShareLock mode, blocked by p8470 of dtx3430845286 con1088328"",""p8453 of dtx3430845280 con1088322 waits for a transactionid lock on ShareLock mode, blocked by p8478 of dtx3430845289 con1088322"",""p8457 of dtx3430845281 con1088323 waits for a transactionid lock on ShareLock mode, blocked by p8450 of dtx3430845278 con1088323""],""seg2"": [""p26938 of dtx3430845281 con1088323 waits for a transactionid lock on ShareLock mode, blocked by p26933 of dtx3430845279 con1088323"",""p26959 of dtx3430845289 con1088328 waits for a transactionid lock on ShareLock mode, blocked by p26952 of dtx3430845286 con1088328"",""p26952 of dtx3430845286 con1088326 waits for a transactionid lock on ShareLock mode, blocked by p26933 of dtx3430845279 con1088326"",""p26947 of dtx3430845285 con1088325 waits for a transactionid lock on ShareLock mode, blocked by p26938 of dtx3430845281 con1088325"",""p26932 of dtx3430845278 con1088320 waits for a transactionid lock on ShareLock mode, blocked by p26941 of dtx3430845283 con1088320""]}",,,,,,,0,,"gddbackend.c",228,
[postgres@mdw log]$
[postgres@mdw log]$
[postgres@mdw log]$
[postgres@mdw log]$
[postgres@mdw log]$
[postgres@mdw log]$ cat gpdb-2024-04-18_000000.csv | grep 'these gxids will be cancelled to break global deadlock'
[postgres@mdw log]$
[postgres@mdw log]$
[postgres@mdw log]$ cat gpdb-2024-04-18_000000.csv | grep 'cancelled by global deadlock detector'
It seems that a deadlock is detected but there is no canceling process
Is it possible to have a core file of the process that have huge amount RSS?
No major changes in disk usage were observed. If the core file is generated, where will it be? I don’t see any core files
If the core file is generated, where will it be? I don’t see any core files
- need set
ulimitto allow generate core - core file location is recorded in
/proc/sys/kernel/core_pattern
$ ulimit -a core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 506905 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 524288 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 131072 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
$ cat /proc/sys/kernel/core_pattern core
There seems to be no limit to core generation
$ coredumpctl list No coredumps found.
It seems that the system cannot find the core file
$ cat /proc/sys/kernel/core_pattern core
Might go to COORDINATOR_DATA_DIRECTORY to see if any?
Next time when it happens, you can help use gcore to generate a core of the GDD process.
$ ls backup_label.old global gpssh.conf pg_commit_ts pg_hba.conf pg_multixact pg_serial pg_stat_tmp pg_twophase pg_xact postmaster.opts base gpbackup_history.db internal.auto.conf pg_distributedlog pg_ident.conf pg_notify pg_snapshots pg_subtrans PG_VERSION postgresql.auto.conf postmaster.pid current_logfiles gpsegconfig_dump log pg_dynshmem pg_logical pg_replslot pg_stat pg_tblspc pg_wal postgresql.conf
There are no coredump files in the COORDINATOR_DATA_DIRECTORY directory.
2024-04-18 02:12:15.032553 UTC,,,p2953,th-1525422016,,,,0,,,seg-1,,,,,"LOG","00000","3rd party error log:
PrivateRefCount: 8192 total in 1 blocks; 2584 free (0 chunks); 5608 used
MdSmgr: 8192 total in 1 blocks; 7656 free (0 chunks); 536 used
LOCALLOCK hash: 8192 total in 1 blocks; 520 free (0 chunks); 7672 used
Timezones: 104120 total in 2 blocks; 2584 free (0 chunks); 101536 used
ErrorContext: 8192 total in 1 blocks; 7896 free (1 chunks); 296 used
Grand total: 67009970944 bytes in 8185 blocks; 351552 free (143 chunks); 67009619392 used
",,,,,,,,"SysLoggerMain","syslogger.c",719,
2024-04-18 02:12:15.046290 UTC,"db","user",p20740,th-1525422016,"172.29.100.121","37591",2024-04-18 00:46:28 UTC,0,con1081112,cmd1271630,seg-1,,,,,"LOG","00000","One or more query execution processes ran out of memory on this segment. Logging memory usage.",,,,,,,0,,,,
The log contains a lot of memory information, is it useful for troubleshooting problems?
Next time when it happens, you can help use gcore to generate a core of the GDD process.
Thank you for your suggestion.
cat gpdb-2024-04-18_000000.csv | grep 'GddContext'
GddContext: 67008200704 total in 7998 blocks; 90592 free (7 chunks); 67008110112 used
GddContext: 67209527296 total in 8022 blocks; 90848 free (7 chunks); 67209436448 used
GddContext: 67301801984 total in 8033 blocks; 90952 free (6 chunks); 67301711032 used
GddContext: 67494739968 total in 8056 blocks; 91192 free (6 chunks); 67494648776 used
GddContext: 67788341248 total in 8091 blocks; 91584 free (7 chunks); 67788249664 used
GddContext: 68484595712 total in 8174 blocks; 92456 free (6 chunks); 68484503256 used
GddContext: 68669145088 total in 8196 blocks; 92704 free (7 chunks); 68669052384 used
GddContext: 69029855232 total in 8239 blocks; 93144 free (6 chunks); 69029762088 used
GddContext: 69893881856 total in 8342 blocks; 94248 free (6 chunks); 69893787608 used
GddContext: 69986156544 total in 8353 blocks; 94360 free (6 chunks); 69986062184 used
GddContext: 71512883200 total in 8535 blocks; 96320 free (7 chunks); 71512786880 used
GddContext: 71722598400 total in 8560 blocks; 96568 free (6 chunks); 71722501832 used
GddContext: 71756152832 total in 8564 blocks; 96616 free (6 chunks); 71756056216 used
GddContext: 71806484480 total in 8570 blocks; 96680 free (6 chunks); 71806387800 used
GddContext: 71814873088 total in 8571 blocks; 96704 free (7 chunks); 71814776384 used
GddContext: 71865204736 total in 8577 blocks; 96768 free (7 chunks); 71865107968 used
GddContext: 72083308544 total in 8603 blocks; 97032 free (6 chunks); 72083211512 used
GddContext: 72175583232 total in 8614 blocks; 97144 free (6 chunks); 72175486088 used
GddContext: 72225914880 total in 8620 blocks; 97208 free (6 chunks); 72225817672 used
GddContext: 72276246528 total in 8626 blocks; 97272 free (6 chunks); 72276149256 used
GddContext: 72318189568 total in 8631 blocks; 97344 free (7 chunks); 72318092224 used
GddContext: 72368521216 total in 8637 blocks; 97408 free (7 chunks); 72368423808 used
GddContext: 72469184512 total in 8649 blocks; 97536 free (7 chunks); 72469086976 used
GddContext: 72485961728 total in 8651 blocks; 97544 free (6 chunks); 72485864184 used
GddContext: 72519516160 total in 8655 blocks; 97600 free (7 chunks); 72519418560 used
GddContext: 72628568064 total in 8668 blocks; 97720 free (6 chunks); 72628470344 used
GddContext: 72636956672 total in 8669 blocks; 97736 free (6 chunks); 72636858936 used
GddContext: 72678899712 total in 8674 blocks; 97784 free (6 chunks); 72678801928 used
GddContext: 72704065536 total in 8677 blocks; 97816 free (6 chunks); 72703967720 used
GddContext: 72737619968 total in 8681 blocks; 97864 free (6 chunks); 72737522104 used
GddContext: 72863449088 total in 8696 blocks; 98024 free (6 chunks); 72863351064 used
GddContext: 72888614912 total in 8699 blocks; 98056 free (6 chunks); 72888516856 used
GddContext: 73131884544 total in 8728 blocks; 98360 free (6 chunks); 73131786184 used
GddContext: 73232547840 total in 8740 blocks; 98488 free (6 chunks); 73232449352 used
GddContext: 73274490880 total in 8745 blocks; 98560 free (7 chunks); 73274392320 used
GddContext: 73324822528 total in 8751 blocks; 98624 free (7 chunks); 73324723904 used
GddContext: 73417097216 total in 8762 blocks; 98728 free (6 chunks); 73416998488 used
GddContext: 73425485824 total in 8763 blocks; 98752 free (7 chunks); 73425387072 used
GddContext: 73450651648 total in 8766 blocks; 98784 free (7 chunks); 73450552864 used
GddContext: 73500983296 total in 8772 blocks; 98848 free (7 chunks); 73500884448 used
GddContext: 73526149120 total in 8775 blocks; 98880 free (7 chunks); 73526050240 used
GddContext: 73568092160 total in 8780 blocks; 98920 free (6 chunks); 73567993240 used
GddContext: 73719087104 total in 8798 blocks; 99112 free (6 chunks); 73718987992 used
GddContext: 73761030144 total in 8803 blocks; 99160 free (6 chunks); 73760930984 used
GddContext: 73928802304 total in 8823 blocks; 99392 free (7 chunks); 73928702912 used
GddContext: 74021076992 total in 8834 blocks; 99496 free (6 chunks); 74020977496 used
GddContext: 74188849152 total in 8854 blocks; 99704 free (6 chunks); 74188749448 used
GddContext: 74356621312 total in 8874 blocks; 99936 free (7 chunks); 74356521376 used
GddContext: 74398564352 total in 8879 blocks; 99976 free (6 chunks); 74398464376 used
GddContext: 74448896000 total in 8885 blocks; 100040 free (6 chunks); 74448795960 used
GddContext: 74457284608 total in 8886 blocks; 100064 free (7 chunks); 74457184544 used
GddContext: 74499227648 total in 8891 blocks; 100104 free (6 chunks); 74499127544 used
GddContext: 74507616256 total in 8892 blocks; 100128 free (7 chunks); 74507516128 used
GddContext: 74532782080 total in 8895 blocks; 100160 free (7 chunks); 74532681920 used
GddContext: 74927046656 total in 8942 blocks; 100648 free (6 chunks); 74926946008 used
GddContext: 74977378304 total in 8948 blocks; 100712 free (6 chunks); 74977277592 used
GddContext: 75052875776 total in 8957 blocks; 100808 free (6 chunks); 75052774968 used
GddContext: 75245813760 total in 8980 blocks; 101048 free (6 chunks); 75245712712 used
GddContext: 75346477056 total in 8992 blocks; 101176 free (6 chunks); 75346375880 used
GddContext: 75438751744 total in 9003 blocks; 101312 free (7 chunks); 75438650432 used
GddContext: 75497472000 total in 9010 blocks; 101368 free (6 chunks); 75497370632 used
GddContext: 75564580864 total in 9018 blocks; 101472 free (7 chunks); 75564479392 used
GddContext: 75623301120 total in 9025 blocks; 101528 free (6 chunks); 75623199592 used
GddContext: 75656855552 total in 9029 blocks; 101576 free (6 chunks); 75656753976 used
GddContext: 75774296064 total in 9043 blocks; 101720 free (6 chunks); 75774194344 used
GddContext: 75782684672 total in 9044 blocks; 101736 free (6 chunks); 75782582936 used
GddContext: 75816239104 total in 9048 blocks; 101792 free (7 chunks); 75816137312 used
GddContext: 75925291008 total in 9061 blocks; 101912 free (6 chunks); 75925189096 used
GddContext: 76151783424 total in 9088 blocks; 102200 free (6 chunks); 76151681224 used
GddContext: 76168560640 total in 9090 blocks; 102240 free (7 chunks); 76168458400 used
GddContext: 76176949248 total in 9091 blocks; 102232 free (6 chunks); 76176847016 used
GddContext: 76185337856 total in 9092 blocks; 102248 free (6 chunks); 76185235608 used
GddContext: 76210503680 total in 9095 blocks; 102280 free (6 chunks); 76210401400 used
GddContext: 76319555584 total in 9108 blocks; 102432 free (7 chunks); 76319453152 used
GddContext: 76369887232 total in 9114 blocks; 102496 free (7 chunks); 76369784736 used
GddContext: 76395053056 total in 9117 blocks; 102528 free (7 chunks); 76394950528 used
GddContext: 76420218880 total in 9120 blocks; 102560 free (7 chunks); 76420116320 used
GddContext: 76453773312 total in 9124 blocks; 102584 free (6 chunks); 76453670728 used
GddContext: 76462161920 total in 9125 blocks; 102600 free (6 chunks); 76462059320 used
GddContext: 76537659392 total in 9134 blocks; 102696 free (6 chunks); 76537556696 used
GddContext: 76562825216 total in 9137 blocks; 102728 free (6 chunks); 76562722488 used
GddContext: 76579602432 total in 9139 blocks; 102744 free (6 chunks); 76579499688 used
GddContext: 76638322688 total in 9146 blocks; 102824 free (6 chunks); 76638219864 used
GddContext: 76713820160 total in 9155 blocks; 102920 free (6 chunks); 76713717240 used
GddContext: 76722208768 total in 9156 blocks; 102944 free (7 chunks); 76722105824 used
GddContext: 76789317632 total in 9164 blocks; 103016 free (6 chunks); 76789214616 used
GddContext: 76822872064 total in 9168 blocks; 103072 free (7 chunks); 76822768992 used
GddContext: 76831260672 total in 9169 blocks; 103064 free (6 chunks); 76831157608 used
GddContext: 76889980928 total in 9176 blocks; 103144 free (6 chunks); 76889877784 used
GddContext: 77200359424 total in 9213 blocks; 103552 free (7 chunks); 77200255872 used
GddContext: 77275856896 total in 9222 blocks; 103648 free (7 chunks); 77275753248 used
GddContext: 77317799936 total in 9227 blocks; 103688 free (6 chunks); 77317696248 used
GddContext: 77326188544 total in 9228 blocks; 103712 free (7 chunks); 77326084832 used
GddContext: 77460406272 total in 9244 blocks; 103864 free (6 chunks); 77460302408 used
GddContext: 77561069568 total in 9256 blocks; 103992 free (6 chunks); 77560965576 used
GddContext: 77577846784 total in 9258 blocks; 104032 free (7 chunks); 77577742752 used
GddContext: 77619789824 total in 9263 blocks; 104072 free (6 chunks); 77619685752 used
GddContext: 77737230336 total in 9277 blocks; 104216 free (6 chunks); 77737126120 used
GddContext: 77745618944 total in 9278 blocks; 104232 free (6 chunks); 77745514712 used
GddContext: 77754007552 total in 9279 blocks; 104256 free (7 chunks); 77753903296 used
GddContext: 77779173376 total in 9282 blocks; 104288 free (7 chunks); 77779069088 used
GddContext: 77795950592 total in 9284 blocks; 104296 free (6 chunks); 77795846296 used
GddContext: 77888225280 total in 9295 blocks; 104408 free (6 chunks); 77888120872 used
GddContext: 77896613888 total in 9296 blocks; 104424 free (6 chunks); 77896509464 used
GddContext: 77905002496 total in 9297 blocks; 104448 free (7 chunks); 77904898048 used
GddContext: 77938556928 total in 9301 blocks; 104472 free (6 chunks); 77938452456 used
GddContext: 77946945536 total in 9302 blocks; 104488 free (6 chunks); 77946841048 used
GddContext: 77980499968 total in 9306 blocks; 104544 free (7 chunks); 77980395424 used
GddContext: 77988888576 total in 9307 blocks; 104536 free (6 chunks); 77988784040 used
GddContext: 77997277184 total in 9308 blocks; 104552 free (6 chunks); 77997172632 used
GddContext: 78005665792 total in 9309 blocks; 104576 free (7 chunks); 78005561216 used
GddContext: 78014054400 total in 9310 blocks; 104568 free (6 chunks); 78013949832 used
GddContext: 78047608832 total in 9314 blocks; 104616 free (6 chunks); 78047504216 used
GddContext: 78106329088 total in 9321 blocks; 104704 free (7 chunks); 78106224384 used
GddContext: 78114717696 total in 9322 blocks; 104696 free (6 chunks); 78114613000 used
GddContext: 78123106304 total in 9323 blocks; 104712 free (6 chunks); 78123001592 used
GddContext: 78131494912 total in 9324 blocks; 104736 free (7 chunks); 78131390176 used
GddContext: 78139883520 total in 9325 blocks; 104728 free (6 chunks); 78139778792 used
GddContext: 78148272128 total in 9326 blocks; 104744 free (6 chunks); 78148167384 used
GddContext: 78232158208 total in 9336 blocks; 104864 free (7 chunks); 78232053344 used
GddContext: 78324432896 total in 9347 blocks; 104968 free (6 chunks); 78324327928 used
GddContext: 78458650624 total in 9363 blocks; 105152 free (7 chunks); 78458545472 used
GddContext: 78483816448 total in 9366 blocks; 105184 free (7 chunks); 78483711264 used
GddContext: 78500593664 total in 9368 blocks; 105192 free (6 chunks); 78500488472 used
GddContext: 78559313920 total in 9375 blocks; 105280 free (7 chunks); 78559208640 used
GddContext: 78567702528 total in 9376 blocks; 105272 free (6 chunks); 78567597256 used
GddContext: 78601256960 total in 9380 blocks; 105320 free (6 chunks); 78601151640 used
GddContext: 78609645568 total in 9381 blocks; 105344 free (7 chunks); 78609540224 used
GddContext: 78659977216 total in 9387 blocks; 105408 free (7 chunks); 78659871808 used
GddContext: 78668365824 total in 9388 blocks; 105400 free (6 chunks); 78668260424 used
GddContext: 78676754432 total in 9389 blocks; 105416 free (6 chunks); 78676649016 used
GddContext: 78701920256 total in 9392 blocks; 105448 free (6 chunks); 78701814808 used
GddContext: 78718697472 total in 9394 blocks; 105464 free (6 chunks); 78718592008 used
GddContext: 78735474688 total in 9396 blocks; 105504 free (7 chunks); 78735369184 used
GddContext: 78769029120 total in 9400 blocks; 105528 free (6 chunks); 78768923592 used
GddContext: 78802583552 total in 9404 blocks; 105576 free (6 chunks); 78802477976 used
GddContext: 78827749376 total in 9407 blocks; 105608 free (6 chunks); 78827643768 used
GddContext: 78844526592 total in 9409 blocks; 105624 free (6 chunks); 78844420968 used
GddContext: 78852915200 total in 9410 blocks; 105640 free (6 chunks); 78852809560 used
GddContext: 78861303808 total in 9411 blocks; 105664 free (7 chunks); 78861198144 used
GddContext: 78936801280 total in 9420 blocks; 105760 free (7 chunks); 78936695520 used
GddContext: 78970355712 total in 9424 blocks; 105784 free (6 chunks); 78970249928 used
GddContext: 79003910144 total in 9428 blocks; 105832 free (6 chunks); 79003804312 used
GddContext: 79045853184 total in 9433 blocks; 105880 free (6 chunks); 79045747304 used
GddContext: 79054241792 total in 9434 blocks; 105896 free (6 chunks); 79054135896 used
GddContext: 79104573440 total in 9440 blocks; 105960 free (6 chunks); 79104467480 used
GddContext: 79171682304 total in 9448 blocks; 106040 free (6 chunks); 79171576264 used
GddContext: 79238791168 total in 9456 blocks; 106144 free (7 chunks); 79238685024 used
GddContext: 79247179776 total in 9457 blocks; 106136 free (6 chunks); 79247073640 used
GddContext: 79289122816 total in 9462 blocks; 106208 free (7 chunks); 79289016608 used
GddContext: 79297511424 total in 9463 blocks; 106200 free (6 chunks); 79297405224 used
GddContext: 79398174720 total in 9475 blocks; 106328 free (6 chunks); 79398068392 used
GddContext: 79465283584 total in 9483 blocks; 106432 free (7 chunks); 79465177152 used
GddContext: 79498838016 total in 9487 blocks; 106456 free (6 chunks); 79498731560 used
GddContext: 79532392448 total in 9491 blocks; 106504 free (6 chunks); 79532285944 used
GddContext: 79540781056 total in 9492 blocks; 106528 free (7 chunks); 79540674528 used
GddContext: 79591112704 total in 9498 blocks; 106592 free (7 chunks); 79591006112 used
GddContext: 79599501312 total in 9499 blocks; 106584 free (6 chunks); 79599394728 used
GddContext: 79633055744 total in 9503 blocks; 106632 free (6 chunks); 79632949112 used
GddContext: 79649832960 total in 9505 blocks; 106648 free (6 chunks); 79649726312 used
GddContext: 79658221568 total in 9506 blocks; 106664 free (6 chunks); 79658114904 used
GddContext: 79674998784 total in 9508 blocks; 106680 free (6 chunks); 79674892104 used
GddContext: 79733719040 total in 9515 blocks; 106760 free (6 chunks); 79733612280 used
GddContext: 79742107648 total in 9516 blocks; 106784 free (7 chunks); 79742000864 used
GddContext: 79750496256 total in 9517 blocks; 106776 free (6 chunks); 79750389480 used
GddContext: 79792439296 total in 9522 blocks; 106848 free (7 chunks); 79792332448 used
GddContext: 79817605120 total in 9525 blocks; 106880 free (7 chunks); 79817498240 used
GddContext: 79867936768 total in 9531 blocks; 106944 free (7 chunks); 79867829824 used
GddContext: 79884713984 total in 9533 blocks; 106952 free (6 chunks); 79884607032 used
GddContext: 79909879808 total in 9536 blocks; 106984 free (6 chunks); 79909772824 used
GddContext: 79918268416 total in 9537 blocks; 107008 free (7 chunks); 79918161408 used
GddContext: 79960211456 total in 9542 blocks; 107048 free (6 chunks); 79960104408 used
GddContext: 79968600064 total in 9543 blocks; 107072 free (7 chunks); 79968492992 used
GddContext: 79976988672 total in 9544 blocks; 107064 free (6 chunks); 79976881608 used
GddContext: 79993765888 total in 9546 blocks; 107104 free (7 chunks); 79993658784 used
GddContext: 80027320320 total in 9550 blocks; 107128 free (6 chunks); 80027213192 used
GddContext: 80044097536 total in 9552 blocks; 107168 free (7 chunks); 80043990368 used
GddContext: 80052486144 total in 9553 blocks; 107160 free (6 chunks); 80052378984 used
GddContext: 80077651968 total in 9556 blocks; 107192 free (6 chunks); 80077544776 used
GddContext: 80086040576 total in 9557 blocks; 107208 free (6 chunks); 80085933368 used
GddContext: 80178315264 total in 9568 blocks; 107320 free (6 chunks); 80178207944 used
GddContext: 80262201344 total in 9578 blocks; 107432 free (6 chunks); 80262093912 used
GddContext: 80379641856 total in 9592 blocks; 107576 free (6 chunks); 80379534280 used
GddContext: 80421584896 total in 9597 blocks; 107648 free (7 chunks); 80421477248 used
GddContext: 80455139328 total in 9601 blocks; 107672 free (6 chunks); 80455031656 used
GddContext: 80530636800 total in 9610 blocks; 107768 free (6 chunks); 80530529032 used
GddContext: 80564191232 total in 9614 blocks; 107816 free (6 chunks); 80564083416 used
GddContext: 80580968448 total in 9616 blocks; 107832 free (6 chunks); 80580860616 used
GddContext: 80622911488 total in 9621 blocks; 107904 free (7 chunks); 80622803584 used
GddContext: 80673243136 total in 9627 blocks; 107968 free (7 chunks); 80673135168 used
GddContext: 80715186176 total in 9632 blocks; 108008 free (6 chunks); 80715078168 used
GddContext: 80740352000 total in 9635 blocks; 108040 free (6 chunks); 80740243960 used
GddContext: 80790683648 total in 9641 blocks; 108104 free (6 chunks); 80790575544 used
GddContext: 80841015296 total in 9647 blocks; 108168 free (6 chunks); 80840907128 used
GddContext: 80941678592 total in 9659 blocks; 108296 free (6 chunks); 80941570296 used
GddContext: 80958455808 total in 9661 blocks; 108312 free (6 chunks); 80958347496 used
GddContext: 80966844416 total in 9662 blocks; 108328 free (6 chunks); 80966736088 used
GddContext: 80983621632 total in 9664 blocks; 108344 free (6 chunks); 80983513288 used
GddContext: 81000398848 total in 9666 blocks; 108384 free (7 chunks); 81000290464 used
GddContext: 81075896320 total in 9675 blocks; 108480 free (7 chunks); 81075787840 used
GddContext: 81109450752 total in 9679 blocks; 108504 free (6 chunks); 81109342248 used
GddContext: 81117839360 total in 9680 blocks; 108520 free (6 chunks); 81117730840 used
GddContext: 81176559616 total in 9687 blocks; 108608 free (7 chunks); 81176451008 used
GddContext: 81210114048 total in 9691 blocks; 108632 free (6 chunks); 81210005416 used
GddContext: 81218502656 total in 9692 blocks; 108648 free (6 chunks); 81218394008 used
GddContext: 81226891264 total in 9693 blocks; 108672 free (7 chunks); 81226782592 used
GddContext: 81243668480 total in 9695 blocks; 108680 free (6 chunks); 81243559800 used
GddContext: 81260445696 total in 9697 blocks; 108696 free (6 chunks); 81260337000 used
GddContext: 81319165952 total in 9704 blocks; 108776 free (6 chunks); 81319057176 used
GddContext: 81327554560 total in 9705 blocks; 108800 free (7 chunks); 81327445760 used
GddContext: 81335943168 total in 9706 blocks; 108792 free (6 chunks); 81335834376 used
GddContext: 81344331776 total in 9707 blocks; 108808 free (6 chunks); 81344222968 used
GddContext: 81377886208 total in 9711 blocks; 108864 free (7 chunks); 81377777344 used
GddContext: 81386274816 total in 9712 blocks; 108856 free (6 chunks); 81386165960 used
GddContext: 81419829248 total in 9716 blocks; 108904 free (6 chunks); 81419720344 used
GddContext: 81428217856 total in 9717 blocks; 108928 free (7 chunks); 81428108928 used
GddContext: 81461772288 total in 9721 blocks; 108952 free (6 chunks); 81461663336 used
GddContext: 81470160896 total in 9722 blocks; 108968 free (6 chunks); 81470051928 used
GddContext: 81478549504 total in 9723 blocks; 108992 free (7 chunks); 81478440512 used
GddContext: 81570824192 total in 9734 blocks; 109096 free (6 chunks); 81570715096 used
GddContext: 81587601408 total in 9736 blocks; 109112 free (6 chunks); 81587492296 used
GddContext: 81637933056 total in 9742 blocks; 109176 free (6 chunks); 81637823880 used
GddContext: 81705041920 total in 9750 blocks; 109280 free (7 chunks); 81704932640 used
GddContext: 81746984960 total in 9755 blocks; 109320 free (6 chunks); 81746875640 used
GddContext: 81847648256 total in 9767 blocks; 109448 free (6 chunks); 81847538808 used
GddContext: 81889591296 total in 9772 blocks; 109496 free (6 chunks); 81889481800 used
GddContext: 81923145728 total in 9776 blocks; 109544 free (6 chunks); 81923036184 used
GddContext: 81939922944 total in 9778 blocks; 109560 free (6 chunks); 81939813384 used
GddContext: 81956700160 total in 9780 blocks; 109600 free (7 chunks); 81956590560 used
GddContext: 81965088768 total in 9781 blocks; 109592 free (6 chunks); 81964979176 used
GddContext: 82015420416 total in 9787 blocks; 109656 free (6 chunks); 82015310760 used
GddContext: 82048974848 tot",,,,,,,,"SysLoggerMain","syslogger.c",719,
GddContext: 82099306496 total in 9797 blocks; 109768 free (6 chunks); 82099196728 used
GddContext: 82107695104 total in 9798 blocks; 109792 free (7 chunks); 82107585312 used
GddContext: 82124472320 total in 9800 blocks; 109800 free (6 chunks); 82124362520 used
GddContext: 82132860928 total in 9801 blocks; 109824 free (7 chunks); 82132751104 used
GddContext: 82141249536 total in 9802 blocks; 109816 free (6 chunks); 82141139720 used
GddContext: 82174803968 total in 9806 blocks; 109864 free (6 chunks); 82174694104 used
GddContext: 82233524224 total in 9813 blocks; 109952 free (7 chunks); 82233414272 used
GddContext: 82241912832 total in 9814 blocks; 109944 free (6 chunks); 82241802888 used
GddContext: 82334187520 total in 9825 blocks; 110080 free (7 chunks); 82334077440 used
GddContext: 82350964736 total in 9827 blocks; 110088 free (6 chunks); 82350854648 used
GddContext: 82376130560 total in 9830 blocks; 110120 free (6 chunks); 82376020440 used
GddContext: 82384519168 total in 9831 blocks; 110144 free (7 chunks); 82384409024 used
GddContext: 82401296384 total in 9833 blocks; 110152 free (6 chunks); 82401186232 used
GddContext: 82434850816 total in 9837 blocks; 110208 free (7 chunks); 82434740608 used
GddContext: 82468405248 total in 9841 blocks; 110232 free (6 chunks); 82468295016 used
GddContext: 82510348288 total in 9846 blocks; 110304 free (7 chunks); 82510237984 used
GddContext: 82518736896 total in 9847 blocks; 110296 free (6 chunks); 82518626600 used
GddContext: 82535514112 total in 9849 blocks; 110336 free (7 chunks); 82535403776 used
GddContext: 82552291328 total in 9851 blocks; 110344 free (6 chunks); 82552180984 used
GddContext: 82560679936 total in 9852 blocks; 110368 free (7 chunks); 82560569568 used
GddContext: 82569068544 total in 9853 blocks; 110360 free (6 chunks); 82568958184 used
GddContext: 82577",,,,,,,,"SysLoggerMain","syslogger.c",719,
GddContext: 82602622976 total in 9857 blocks; 110408 free (6 chunks); 82602512568 used
GddContext: 82652954624 total in 9863 blocks; 110472 free (6 chunks); 82652844152 used
GddContext: 82686509056 total in 9867 blocks; 110528 free (7 chunks); 82686398528 used
GddContext: 82803949568 total in 9881 blocks; 110664 free (6 chunks); 82803838904 used
GddContext: 82812338176 total in 9882 blocks; 110688 free (7 chunks); 82812227488 used
GddContext: 82820726784 total in 9883 blocks; 110680 free (6 chunks); 82820616104 used
GddContext: 82887835648 total in 9891 blocks; 110784 free (7 chunks); 82887724864 used
GddContext: 82896224256 total in 9892 blocks; 110776 free (6 chunks); 82896113480 used
GddContext: 82921390080 total in 9895 blocks; 110808 free (6 chunks); 82921279272 used
GddContext: 82980110336 total in 9902 blocks; 110888 free (6 chunks); 82979999448 used
GddContext: 82988498944 total in 9903 blocks; 110912 free (7 chunks); 82988388032 used
GddContext: 83005276160 total in 9905 blocks; 110920 free (6 chunks); 83005165240 used
GddContext: 83022053376 total in 9907 blocks; 110936 free (6 chunks); 83021942440 used
GddContext: 83047219200 total in 9910 blocks; 110968 free (6 chunks); 83047108232 used
GddContext: 83063996416 total in 9912 blocks; 111008 free (7 chunks); 83063885408 used
GddContext: 83097550848 total in 9916 blocks; 111032 free (6 chunks); 83097439816 used
GddContext: 83105939456 total in 9917 blocks; 111048 free (6 chunks); 83105828408 used
GddContext: 83131105280 total in 9920 blocks; 111080 free (6 chunks); 83130994200 used
GddContext: 83189825536 total in 9927 blocks; 111168 free (7 chunks); 83189714368 used
GddContext: 83206602752 total in 9929 blocks; 111176 free (6 chunks); 83206491576 used
GddContext: 83240157184 total in 9933 blocks; 111232 free (7 chunks); 83240045952 used
GddContext: 83256934400 total in 9935 blocks; 111240 free (6 chunks); 83256823160 used
GddContext: 83265323008 total in 9936 blocks; 111264 free (7 chunks); 83265211744 used
GddContext: 83273711616 total in 9937 blocks; 111256 free (6 chunks); 83273600360 used
GddContext: 83298877440 total in 9940 blocks; 111288 free (6 chunks); 83298766152 used
GddContext: 83307266048 total in 9941 blocks; 111304 free (6 chunks); 83307154744 used
GddContext: 83315654656 total in 9942 blocks; 111328 free (7 chunks); 83315543328 used
GddContext: 83349209088 total in 9946 blocks; 1",,,,,,,,"SysLoggerMain","syslogger.c",719,
GddContext: 83357597696 total in 9947 blocks; 111368 free (6 chunks); 83357486328 used
GddContext: 83391152128 total in 9951 blocks; 111424 free (7 chunks); 83391040704 used
GddContext: 83399540736 total in 9952 blocks; 111416 free (6 chunks); 83399429320 used
GddContext: 83407929344 total in 9953 blocks; 111432 free (6 chunks); 83407817912 used
GddContext: 83533758464 total in 9968 blocks; 111592 free (6 chunks); 83533646872 used
GddContext: 83542147072 total in 9969 blocks; 111616 free (7 chunks); 83542035456 used
GddContext: 83575701504 total in 9973 blocks; 111640 free (6 chunks); 83575589864 used
GddContext: 83651198976 total in 9982 blocks; 111736 free (6 chunks); 83651087240 used
GddContext: 83659587584 total in 9983 blocks; 111752 free (6 chunks); 83659475832 used
GddContext: 83709919232 total in 9989 blocks; 111816 free (6 chunks); 83709807416 used
GddContext: 83718307840 total in 9990 blocks; 111840 free (7 chunks); 83718196000 used
GddContext: 83726696448 total in 9991 blocks; 111832 free (6 chunks); 83726584616 used
GddContext: 83735085056 total in 9992 blocks; 111848 free (6 chunks); 83734973208 used
GddContext: 83743473664 total in 9993 blocks; 111872 free (7 chunks); 83743361792 used
GddContext: 83751862272 total in 9994 blocks; 111864 free (6 chunks); 83751750408 used
GddContext: 83760250880 total in 9995 blocks; 111880 free (6 chunks); 83760139000 used
GddContext: 83785416704 total in 9998 blocks; 111912 free (6 chunks); 83785304792 used
GddContext: 83793805312 total in 9999 blocks; 111936 free (7 chunks); 83793693376 used
GddContext: 83852525568 total in 10006 blocks; 111992 free (6 chunks); 83852413576 used
GddContext: 83860914176 total in 10007 blocks; 112008 free (6 chunks); 83860802168 used
GddContext: 83869302784 total in 10008 blocks; 112032 free (7 chunks); 83869190752 used
GddContext: 83928023040 total in 10015 blocks; 112088 free (6 chunks); 83927910952 used
GddContext: 83936411648 total in 10016 blocks; 112104 free (6 chunks); 83936299544 used
GddContext: 83969966080 total in 10020 blocks; 112160 free (7 chunks); 83969853920 used
GddContext: 84045463552 total in 10029 blocks; 112256 free (7 chunks); 84045351296 used
GddContext: 84053852160 total in 10030 blocks; 112248 free (6 chunks); 84053739912 used
GddContext: 84062240768 total in 10031 blocks; 112264 free (6 chunks); 84062128504 used
GddContext: 84112572416 total in 10037 blocks; 112328 free (6 chunks); 84112460088 used
GddContext: 84120961024 total in 10038 blocks; 112352 free (7 chunks); 84120848672 used
GddContext: 84246790144 total in 10053 blocks; 112512 free (7 chunks); 84246677632 used
GddContext: 84255178752 total in 10054 blocks; 112504 free (6 chunks); 84255066248 used
GddContext: 84322287616 total in 10062 blocks; 11",,,,,,,,"SysLoggerMain","syslogger.c",719,
GddContext: 84330676224 total in 10063 blocks; 112600 free (6 chunks); 84330563624 used
GddContext: 84339064832 total in 10064 blocks; 112616 free (6 chunks); 84338952216 used
GddContext: 84347453440 total in 10065 blocks; 112640 free (7 chunks); 84347340800 used
GddContext: 84422950912 total in 10074 blocks; 112736 free (7 chunks); 84422838176 used
GddContext: 84481671168 total in 10081 blocks; 112792 free (6 chunks); 84481558376 used
GddContext: 84490059776 total in 10082 blocks; 112808 free (6 chunks); 84489946968 used
GddContext: 84498448384 total in 10083 blocks; 112832 free (7 chunks); 84498335552 used
GddContext: 84506836992 total in 10084 blocks; 112824 free (6 chunks); 84506724168 used
GddContext: 84515225600 total in 10085 blocks; 112840 free (6 chunks); 84515112760 used
GddContext: 84523614208 total in 10086 blocks; 112864 free (7 chunks); 84523501344 used
GddContext: 84607500288 total in 10096 blocks; 112952 free (6 chunks); 84607387336 used
GddContext: 84615888896 total in 10097 blocks; 112968 free (6 chunks); 84615775928 used
GddContext: 84624277504 total in 10098 blocks; 112992 free (7 chunks); 84624164512 used
GddContext: 84632666112 total in 10099 blocks; 112984 free (6 chunks); 84632553128 used
GddContext: 84641054720 total in 10100 blocks; 113000 free (6 chunks); 84640941720 used
GddContext: 84741718016 total in 10112 blocks; 113128 free (6 chunks); 84741604888 used
GddContext: 84750106624 total in 10113 blocks; 113152 free (7 chunks); 84749993472 used
GddContext: 84758495232 total in 10114 blocks; 113144 free (6 chunks); 84758382088 used
GddContext: 84817215488 total in 10121 blocks; 113224 free (6 chunks); 84817102264 used
GddContext: 84909490176 total in 10132 blocks; 113336 free (6 chunks); 84909376840 used
GddContext: 84917878784 total in 10133 blocks; 113352 free (6 chunks); 84917765432 used
GddContext: 84926267392 total in 10134 blocks; 113376 free (7 chunks); 84926154016 used
GddContext: 84943044608 total in 10136 blocks; 113384 free (6 chunks); 84942931224 used
GddContext: 84951433216 total in 10137 blocks; 113408 free (7 chunks); 84951319808 used
GddContext: 85010153472 total in 10144 blocks; 113464 free (6 chunks); 85010040008 used
GddContext: 85018542080 total in 10145 blocks; 113480 free (6 chunks); 85018428600 used
GddContext: 85026930688 total in 10146 blocks; 113504 free (7 chunks); 85026817184 used
GddContext: 85060485120 total in 10150 blocks; 113528 free (6 chunks); 85060371592 used
GddContext: 85094039552 total in 10154 blocks; 113576 free (6 chunks); 85093925976 used
GddContext: 85102428160 total in 10155 blocks; 113600 free (7 chunks); 85102314560 used
GddContext: 85110816768 total in 10156 blocks; 113592 free (6 chunks); 85110703176 used
GddContext: 85119205376 total in 10157 blocks; 113608 free (6 chunks); 85119091768 used
GddContext: 85177925632 total in 10164 blocks; 113696 free (7 chunks); 85177811936 used
GddContext: 85186314240 total in 10165 blocks; 113688 free (6 chunks); 85186200552 used
GddContext: 85194702848 total in 10166 blocks; 113704 free (6 chunks); 85194589144 used
GddContext: 85203091456 total in 10167 blocks; 113728 free (7 chunks); 85202977728 used
GddContext: 85211480064 total in 10168 blocks; 113720 free (6 chunks); 85211366344 used
GddContext: 85253423104 total in 10173 blocks; 113792 free (7 chunks); 85253309312 used
GddContext: 85303754752 total in 10179 blocks; 113856 free (7 chunks); 85303640896 used
GddContext: 85312143360 total in 10180 blocks; 113848 free (6 chunks); 85312029512 used
GddContext: 85370863616 total in 10187 blocks; 113928 free (6 chunks); 85370749688 used
GddContext: 85379252224 total in 10188 blocks; 113952 free (7 chunks); 85379138272 used
GddContext: 85387640832 total in 10189 blocks; 113944 free (6 chunks); 85387526888 used
GddContext: 85404418048 total in 10191 blocks; 113984 free (7 chunks); 85404304064 used
GddContext: 85496692736 total in 10202 blocks; 114088 free (6 chunks); 85496578648 used
GddContext: 85505081344 total in 10203 blocks; 114112 free (7 chunks); 85504967232 used
GddContext: 85513469952 total in 10204 blocks; 114104 free (6 chunks); 85513355848 used
GddContext: 85538635776 total in 10207 blocks; 114136 free (6 chunks); 85538521640 used
GddContext: 85547024384 total in 10208 blocks; 114152 free (6 chunks); 85546910232 used
GddContext: 85572190208 total in 10211 blocks; 114184 free (6 chunks); 85572076024 used
GddContext: 85580578816 total in 10212 blocks; 114208 free (7 chunks); 85580464608 used
GddContext: 85588967424 total in 10213 blocks; 114200 free (6 chunks); 85588853224 used
GddContext: 85656076288 total in 10221 blocks; 114304 free (7 chunks); 85655961984 used
GddContext: 85664464896 total in 10222 blocks; 114296 free (6 chunks); 85664350600 used
GddContext: 85672853504 total in 10223 blocks; 114312 free (6 chunks); 85672739192 used
GddContext: 85748350976 total in 10232 blocks; 114408 free (6 chunks); 85748236568 used
GddContext: 85823848448 total in 10241 blocks; 114504 free (6 chunks); 85823733944 used
GddContext: 85832237056 total in 10242 blocks; 114528 free (7 chunks); 85832122528 used
GddContext: 85840625664 total in 10243 blocks; 114520 free (6 chunks); 85840511144 used
GddContext: 85874180096 total in 10247 blocks; 114568 free (6 chunks); 85874065528 used
GddContext: 85882568704 total in 10248 blocks; 114592 free (7 chunks); 85882454112 used
GddContext: 85890957312 total in 10249 blocks; 114584 free (6 chunks); 85890842728 used
GddContext: 85932900352 total in 10254 blocks; 114656 free (7 chunks); 85932785696 used
GddContext: 85941288960 total in 10255 blocks; 114648 free (6 chunks); 85941174312 used
GddContext: 85949677568 total in 10256 blocks; 114664 free (6 chunks); 85949562904 used
GddContext: 86000009216 total in 10262 blocks; 114728 free (6 chunks); 85999894488 used
GddContext: 86025175040 total in 10265 blocks; 114760 free (6 chunks); 86025060280 used
GddContext: 86033563648 total in 10266 blocks; 114784 free (7 chunks); 86033448864 used
GddContext: 86041952256 total in 10267 blocks; 114776 free (6 chunks); 86041837480 used
GddContext: 86058729472 total in 10269 blocks; 114816 free (7 chunks); 86058614656 used
GddContext: 86100672512 total in 10274 blocks; 114856 free (6 chunks); 86100557656 used
GddContext: 86125838336 total in 10277 blocks; 114888 free (6 chunks); 86125723448 used
GddContext: 86218113024 total in 10288 blocks; 115000 free (6 chunks); 86217998024 used
GddContext: 86226501632 total in 10289 blocks; 115016 free (6 chunks); 86226386616 used
GddContext: 86251667456 total in 10292 blocks; 115048 free (6 chunks); 86251552408 used
GddContext: 86268444672 total in 10294 blocks; 115064 free (6 chunks); 86268329608 used
GddContext: 86276833280 total in 10295 blocks; 115080 free (6 chunks); 86276718200 used
GddContext: 86301999104 total in 10298 blocks; 115112 free (6 chunks); 86301883992 used
GddContext: 86360719360 total in 10305 blocks; 115200 free (7 chunks); 86360604160 used
GddContext: 86377496576 total in 10307 blocks; 115208 free (6 chunks); 86377381368 used
GddContext: 86394273792 total in 10309 blocks; 115224 free (6 chunks); 86394158568 used
GddContext: 86419439616 total in 10312 blocks; 115256 free (6 chunks); 86419324360 used
GddContext: 86436216832 total in 10314 blocks; 115296 free (7 chunks); 86436101536 used
GddContext: 86486548480 total in 10320 blocks; 115360 free (7 chunks); 86486433120 used
GddContext: 86520102912 total in 10324 blocks; 115384 free (6 chunks); 86519987528 used
GddContext: 86536880128 total in 10326 blocks; 115424 free (7 chunks); 86536764704 used
GddContext: 86545268736 total in 10327 blocks; 115416 free (6 chunks); 86545153320 used
GddContext: 86553657344 total in 10328 blocks; 115432 free (6 chunks); 86553541912 used
GddContext: 86570434560 total in 10330 blocks; 115448 free (6 chunks); 86570319112 used
GddContext: 86629154816 total in 10337 blocks; 115528 free (6 chunks); 86629039288 used
GddContext: 86637543424 total in 10338 blocks; 115552 free (7 chunks); 86637427872 used
GddContext: 86645932032 total in 10339 blocks; 115544 free (6 chunks); 86645816488 used
GddContext: 86654320640 total in 10340 blocks; 115560 free (6 chunks); 86654205080 used
GddContext: 86729818112 total in 10349 blocks; 115656 free (6 chunks); 86729702456 used
GddContext: 86746595328 total in 10351 blocks; 115672 free (6 chunks); 86746479656 used
GddContext: 86754983936 total in 10352 blocks; 115688 free (6 chunks); 86754868248 used
GddContext: 86780149760 total in 10355 blocks; 115720 free (6 chunks); 86780034040 used
GddContext: 86847258624 total in 10363 blocks; 115800 free (6 chunks); 86847142824 used
GddContext: 86855647232 total in 10364 blocks; 115816 free (6 chunks); 86855531416 used
GddContext: 86864035840 total in 10365 blocks; 115840 free (7 chunks); 86863920000 used
GddContext: 86872424448 total in 10366 blocks; 115832 free (6 chunks); 86872308616 used
GddContext: 86897590272 total in 10369 blocks; 115864 free (6 chunks); 86897474408 used
GddContext: 86905978880 total in 10370 blocks; 115880 free (6 chunks); 86905863000 used
GddContext: 86914367488 total in 10371 blocks; 115904 free (7 chunks); 86914251584 used
GddContext: 86956310528 total in 10376 blocks; 115944 free (6 chunks); 86956194584 used
GddContext: 86964699136 total in 10377 blocks; 115968 free (7 chunks); 86964583168 used
GddContext: 86973087744 total in 10378 blocks; 115960 free (6 chunks); 86972971784 used
GddContext: 86981476352 total in 10379 blocks; 115976 free (6 chunks); 86981360376 used
GddContext: 87040196608 total in 10386 blocks; 116064 free (7 chunks); 87040080544 used
GddContext: 87048585216 total in 10387 blocks; 116056 free (6 chunks); 87048469160 used
GddContext: 87056973824 total in 10388 blocks; 116072 free (6 chunks); 87056857752 used
GddContext: 87065362432 total in 10389 blocks; 116096 free (7 chunks); 87065246336 used
GddContext: 87073751040 total in 10390 blocks; 116088 free (6 chunks); 87073634952 used
GddContext: 87107305472 total in 10394 blocks; 116136 free (6 chunks); 87107189336 used
GddContext: 87124082688 total in 10396 blocks; 116152 free (6 chunks); 87123966536 used
GddContext: 87132471296 total in 10397 blocks; 116168 free (6 chunks); 87132355128 used
GddContext: 87166025728 total in 10401 blocks; 116224 free (7 chunks); 87165909504 used
GddContext: 87199580160 total in 10405 blocks; 116248 free (6 chunks); 87199463912 used
GddContext: 87207968768 total in 10406 blocks; 116264 free (6 chunks); 87207852504 used
GddContext: 87216357376 total in 10407 blocks; 116288 free (7 chunks); 87216241088 used
GddContext: 87266689024 total in 10413 blocks; 116352 free (7 chunks); 87266572672 used
GddContext: 87275077632 total in 10414 blocks; 116344 free (6 chunks); 87274961288 used
GddContext: 87283466240 total in 10415 blocks; 116360 free (6 chunks); 87283349880 used
GddContext: 87291854848 total in 10416 blocks; 116384 free (7 chunks); 87291738464 used
GddContext: 87300243456 total in 10417 blocks; 116376 free (6 chunks); 87300127080 used
GddContext: 87342186496 total in 10422 blocks; 116448 free (7 chunks); 87342070048 used
GddContext: 87350575104 total in 10423 blocks; 116440 free (6 chunks); 87350458664 used
GddContext: 87384129536 total in 10427 blocks; 116488 free (6 chunks); 87384013048 used
GddContext: 87392518144 total in 10428 blocks; 116512 free (7 chunks); 87392401632 used
GddContext: 87417683968 total in 10431 blocks; 116544 free (7 chunks); 87417567424 used
GddContext: 87426072576 total in 10432 blocks; 116536 free (6 chunks); 87425956040 used
GddContext: 87476404224 total in 10438 blocks; 116600 free (6 chunks); 87476287624 used
GddContext: 87518347264 total in 10443 blocks; 116672 free (7 chunks); 87518230592 used
GddContext: 87535124480 total in 10445 blocks; 116680 free (6 chunks); 87535007800 used
GddContext: 87560290304 total in 10448 blocks; 116712 free (6 chunks); 87560173592 used
GddContext: 87585456128 total in 10451 blocks; 116744 free (6 chunks); 87585339384 used
GddContext: 87593844736 total in 10452 blocks; 116768 free (7 chunks); 87593727968 used
GddContext: 87602233344 total in 10453 blocks; 116760 free (6 chunks); 87602116584 used
GddContext: 87619010560 total in 10455 blocks; 116800 free (7 chunks); 87618893760 used
GddContext: 87644176384 total in 10458 blocks; 116832 free (7 chunks); 87644059552 used
GddContext: 87677730816 total in 10462 blocks; 116856 free (6 chunks); 87677613960 used
GddContext: 87702896640 total in 10465 blocks; 116888 free (6 chunks); 87702779752 used
GddContext: 87719673856 total in 10467 blocks; 116928 free (7 chunks); 87719556928 used
GddContext: 87786782720 total in 10475 blocks; 117000 free (6 chunks); 87786665720 used
GddContext: 87803559936 total in 10477 blocks; 117016 free (6 chunks); 87803442920 used
GddContext: 87811948544 total in 10478 blocks; 117032 free (6 chunks); 87811831512 used
GddContext: 87853891584 total in 10483 blocks; 117080 free (6 chunks); 87853774504 used
GddContext: 87879057408 total in 10486 blocks; 117112 free (6 chunks); 87878940296 used
GddContext: 87921000448 total in 10491 blocks; 117184 free (7 chunks); 87920883264 used
GddContext: 87954554880 total in 10495 blocks; 117208 free (6 chunks); 87954437672 used
GddContext: 87971332096 total in 10497 blocks; 117248 free (7 chunks); 87971214848 used
GddContext: 88013275136 total in 10502 blocks; 117288 free (6 chunks); 88013157848 used
GddContext: 88063606784 total in 10508 blocks; 117352 free (6 chunks); 88063489432 used
GddContext: 88071995392 total in 10509 blocks; 117376 free (7 chunks); 88071878016 used
GddContext: 88097161216 total in 10512 blocks; 117408 free (7 chunks); 88097043808 used
GddContext: 88105549824 total in 10513 blocks; 117400 free (6 chunks); 88105432424 used
GddContext: 88122327040 total in 10515 blocks; 117440 free (7 chunks); 88122209600 used
GddContext: 88172658688 total in 10521 blocks; 117504 free (7 chunks); 88172541184 used
GddContext: 88181047296 total in 10522 blocks; 117496 free (6 chunks); 88180929800 used
GddContext: 88189435904 total in 10523 blocks; 117512 free (6 chunks); 88189318392 used
GddContext: 88222990336 total in 10527 blocks; 117568 free (7 chunks); 88222872768 used
GddContext: 88231378944 total in 10528 blocks; 117560 free (6 chunks); 88231261384 used
GddContext: 88248156160 total in 10530 blocks; 117600 free (7 chunks); 88248038560 used
GddContext: 88256544768 total in 10531 blocks; 117592 free (6 chunks); 88256427176 used
GddContext: 88273321984 total in 10533 blocks; 117632 free (7 chunks); 88273204352 used
GddContext: 88332042240 total in 10540 blocks; 117688 free (6 chunks); 88331924552 used
GddContext: 88357208064 total in 10543 blocks; 117720 free (6 chunks); 88357090344 used
GddContext: 88399151104 total in 10548 blocks; 117792 free (7 chunks); 88399033312 used
GddContext: 88432705536 total in 10552 blocks; 117816 free (6 chunks); 88432587720 used
GddContext: 88533368832 total in 10564 blocks; 117944 free (6 chunks); 88533250888 used
GddContext: 88541757440 total in 10565 blocks; 117960 free (6 chunks); 88541639480 used
GddContext: 88566923264 total in 10568 blocks; 117992 free (6 chunks); 88566805272 used
GddContext: 88650809344 total in 10578 blocks; 118112 free (7 chunks); 88650691232 used
GddContext: 88675975168 total in 10581 blocks; 118144 free (7 chunks); 88675857024 used
GddContext: 88734695424 total in 10588 blocks; 118200 free (6 chunks); 88734577224 used
GddContext: 88743084032 total in 10589 blocks; 118216 free (6 chunks); 88742965816 used
GddContext: 88751472640 total in 10590 blocks; 118240 free (7 chunks); 88751354400 used
GddContext: 88759861248 total in 10591 blocks; 118232 free (6 chunks); 88759743016 used
GddContext: 88793415680 total in 10595 blocks; 118280 free (6 chunks); 88793297400 used
GddContext: 88826970112 total in 10599 blocks; 118336 free (7 chunks); 88826851776 used
GddContext: 88894078976 total in 10607 blocks; 118408 free (6 chunks); 88893960568 used
GddContext: 88902467584 total in 10608 blocks; 118432 free (7 chunks); 88902349152 used
GddContext: 88944410624 total in 10613 blocks; 118472 free (6 chunks); 88944292152 used
GddContext: 88952799232 total in 10614 blocks; 118496 free (7 chunks); 88952680736 used
GddContext: 88986353664 total in 10618 blocks; 118520 free (6 chunks); 88986235144 used
GddContext: 88994742272 total in 10619 blocks; 118536 free (6 chunks); 88994623736 used
GddContext: 89036685312 total in 10624 blocks; 118584 free (6 chunks); 89036566728 used
GddContext: 89045073920 total in 10625 blocks; 118600 free (6 chunks); 89044955320 used
GddContext: 89053462528 total in 10626 blocks; 118624 free (7 chunks); 89053343904 used
GddContext: 89078628352 total in 10629 blocks; 118656 free (7 chunks); 89078509696 used
GddContext: 89095405568 total in 10631 blocks; 118664 free (6 chunks); 89095286904 used
GddContext: 89137348608 total in 10636 blocks; 118712 free (6 chunks); 89137229896 used
GddContext: 89154125824 total in 10638 blocks; 118752 free (7 chunks); 89154007072 used
GddContext: 89196068864 total in 10643 blocks; 118792 free (6 chunks); 89195950072 used
GddContext: 89229623296 total in 10647 blocks; 118848 free (7 chunks); 89229504448 used
GddContext: 89279954944 total in 10653 blocks; 118912 free (7 chunks); 89279836032 used
GddContext: 89296732160 total in 10655 blocks; 118920 free (6 chunks); 89296613240 used
GddContext: 89321897984 total in 10658 blocks; 118952 free (6 chunks); 89321779032 used
GddContext: 89338675200 total in 10660 blocks; 118968 free (6 chunks); 89338556232 used
GddContext: 89414172672 total in 10669 blocks; 119064 free (6 chunks); 89414053608 used
GddContext: 89422561280 total in 10670 blocks; 119080 free (6 chunks); 89422442200 used
GddContext: 89430949888 total in 10671 blocks; 119104 free (7 chunks); 89430830784 used
GddContext: 89447727104 total in 10673 blocks; 119112 free (6 chunks); 89447607992 used
GddContext: 89506447360 total in 10680 blocks; 119200 free (7 chunks); 89506328160 used
GddContext: 89514835968 total in 10681 blocks; 119192 free (6 chunks); 89514716776 used
GddContext: 89531613184 total in 10683 blocks; 119232 free (7 chunks); 89531493952 used
GddContext: 89556779008 total in 10686 blocks; 119264 free (7 chunks); 89556659744 used
GddContext: 89581944832 total in 10689 blocks; 119296 free (7 chunks); 89581825536 used
GddContext: 89590333440 total in 10690 blocks; 119288 free (6 chunks); 89590214152 used
GddContext: 89598722048 total in 10691 blocks; 119304 free (6 chunks); 89598602744 used
GddContext: 89607110656 total in 10692 blocks; 119328 free (7 chunks); 89606991328 used
GddContext: 89615499264 total in 10693 blocks; 119320 free (6 chunks); 89615379944 used
GddContext: 89623887872 total in 10694 blocks; 119336 free (6 chunks); 89623768536 used
GddContext: 89690996736 total in 10702 blocks; 119416 free (6 chunks); 89690877320 used
GddContext: 89716162560 total in 10705 blocks; 119448 free (6 chunks); 89716043112 used
GddContext: 89724551168 total in 10706 blocks; 119464 free (6 chunks); 89724431704 used
GddContext: 89732939776 total in 10707 blocks; 119488 free (7 chunks); 89732820288 used
GddContext: 89741328384 total in 10708 blocks; 119480 free (6 chunks); 89741208904 used
GddContext: 89749716992 total in 10709 blocks; 119496 free (6 chunks); 89749597496 used
GddContext: 89758105600 total in 10710 blocks; 119520 free (7 chunks); 89757986080 used
GddContext: 89766494208 total in 10711 blocks; 119512 free (6 chunks); 89766374696 used
GddContext: 89800048640 total in 10715 blocks; 119560 free (6 chunks); 89799929080 used
GddContext: 89825214464 total in 10718 blocks; 119592 free (6 chunks); 89825094872 used
GddContext: 89833603072 total in 10719 blocks; 119616 free (7 chunks); 89833483456 used
GddContext: 89850380288 total in 10721 blocks; 119624 free (6 chunks); 89850260664 used
GddContext: 89858768896 total in 10722 blocks; 119648 free (7 chunks); 89858649248 used
GddContext: 89883934720 total in 10725 blocks; 119680 free (7 chunks); 89883815040 used
GddContext: 89900711936 total in 10727 blocks; 119688 free (6 chunks); 89900592248 used
GddContext: 89934266368 total in 10731 blocks; 119744 free (7 chunks); 89934146624 used
GddContext: 89967820800 total in 10735 blocks; 119768 free (6 chunks); 89967701032 used
GddContext: 89984598016 total in 10737 blocks; 119808 free (7 chunks); 89984478208 used
GddContext: 90001375232 total in 10739 blocks; 119816 free (6 chunks); 90001255416 used
GddContext: 90034929664 total in 10743 blocks; 119872 free (7 chunks); 90034809792 used
GddContext: 90043318272 total in 10744 blocks; 119864 free (6 chunks); 90043198408 used
GddContext: 90060095488 total in 10746 blocks; 119904 free (7 chunks); 90059975584 used
GddContext: 90068484096 total in 10747 blocks; 119896 free (6 chunks); 90068364200 used
GddContext: 90102038528 total in 10751 blocks; 119944 free (6 chunks); 90101918584 used
GddContext: 90135592960 total in 10755 blocks; 120000 free (7 chunks); 90135472960 used
GddContext: 90152370176 total in 10757 blocks; 120008 free (6 chunks); 90152250168 used
GddContext: 90169147392 total in 10759 blocks; 120024 free (6 chunks); 90169027368 used
GddContext: 90211090432 total in 10764 blocks; 120096 free (7 chunks); 90210970336 used
GddContext: 90219479040 total in 10765 blocks; 120088 free (6 chunks); 90219358952 used
GddContext: 90236256256 total in 10767 blocks; 120128 free (7 chunks); 90236136128 used
GddContext: 90286587904 total in 10773 blocks; 120192 free (7 chunks); 90286467712 used
GddContext: 90294976512 total in 10774 blocks; 120184 free (6 chunks); 90294856328 used
GddContext: 90345308160 total in 10780 blocks; 120248 free (6 chunks); 90345187912 used
GddContext: 90353696768 total in 10781 blocks; 120264 free (6 chunks); 90353576504 used
GddContext: 90362085376 total in 10782 blocks; 120288 free (7 chunks); 90361965088 used
GddContext: 90387251200 total in 10785 blocks; 120320 free (7 chunks); 90387130880 used
GddContext: 90395639808 total in 10786 blocks; 120312 free (6 chunks); 90395519496 used
GddContext: 90429194240 total in 10790 blocks; 120360 free (6 chunks); 90429073880 used
GddContext: 90445971456 total in 10792 blocks; 120376 free (6 chunks); 90445851080 used
GddContext: 90454360064 total in 10793 blocks; 120392 free (6 chunks); 90454239672 used
GddContext: 90496303104 total in 10798 blocks; 120440 free (6 chunks); 90496182664 used
GddContext: 90504691712 total in 10799 blocks; 120456 free (6 chunks); 90504571256 used
GddContext: 90521468928 total in 10801 blocks; 120472 free (6 chunks); 90521348456 used
GddContext: 90546634752 total in 10804 blocks; 120504 free (6 chunks); 90546514248 used
GddContext: 90555023360 total in 10805 blocks; 120520 free (6 chunks); 90554902840 used
GddContext: 90563411968 total in 10806 blocks; 120544 free (7 chunks); 90563291424 used
GddContext: 90580189184 total in 10808 blocks; 120552 free (6 chunks); 90580068632 used
GddContext: 90588577792 total in 10809 blocks; 120576 free (7 chunks); 90588457216 used
GddContext: 90596966400 total in 10810 blocks; 120568 free (6 chunks); 90596845832 used
GddContext: 90622132224 total in 10813 blocks; 120600 free (6 chunks); 90622011624 used
GddContext: 90630520832 total in 10814 blocks; 120616 free (6 chunks); 90630400216 used
GddContext: 90638909440 total in 10815 blocks; 120640 free (7 chunks); 90638788800 used
GddContext: 90747961344 total in 10828 blocks; 120760 free (6 chunks); 90747840584 used
GddContext: 90756349952 total in 10829 blocks; 120776 free (6 chunks); 90756229176 used
GddContext: 90764738560 total in 10830 blocks; 120800 free (7 chunks); 90764617760 used
GddContext: 90773127168 total in 10831 blocks; 120792 free (6 chunks); 90773006376 used
GddContext: 90831847424 total in 10838 blocks; 120872 free (6 chunks); 90831726552 used
GddContext: 90898956288 total in 10846 blocks; 120952 free (6 chunks); 90898835336 used
GddContext: 90924122112 total in 10849 blocks; 120984 free (6 chunks); 90924001128 used
GddContext: 90932510720 total in 10850 blocks; 121000 free (6 chunks); 90932389720 used
GddContext: 90940899328 total in 10851 blocks; 121024 free (7 chunks); 90940778304 used
GddContext: 90999619584 total in 10858 blocks; 121080 free (6 chunks); 90999498504 used
GddContext: 91008008192 total in 10859 blocks; 121096 free (6 chunks); 91007887096 used
GddContext: 91016396800 total in 10860 blocks; 121120 free (7 chunks); 91016275680 used
GddContext: 91024785408 total in 10861 blocks; 121112 free (6 chunks); 91024664296 used
GddContext: 91066728448 total in 10866 blocks; 121184 free (7 chunks); 91066607264 used
GddContext: 91075117056 total in 10867 blocks; 121176 free (6 chunks); 91074995880 used
GddContext: 91159003136 total in 10877 blocks; 121288 free (6 chunks); 91158881848 used
GddContext: 91167391744 total in 10878 blocks; 121312 free (7 chunks); 91167270432 used
GddContext: 91251277824 total in 10888 blocks; 121400 free (6 chunks); 91251156424 used
GddContext: 91259666432 total in 10889 blocks; 121416 free (6 chunks); 91259545016 used
GddContext: 91268055040 total in 10890 blocks; 121440 free (7 chunks); 91267933600 used
GddContext: 91276443648 total in 10891 blocks; 121432 free (6 chunks); 91276322216 used
GddContext: 91318386688 total in 10896 blocks; 121504 free (7 chunks); 91318265184 used
GddContext: 91326775296 total in 10897 blocks; 121496 free (6 chunks); 91326653800 used
GddContext: 91335163904 total in 10898 blocks; 121512 free (6 chunks); 91335042392 used
GddContext: 91343552512 total in 10899 blocks; 121536 free (7 chunks); 91343430976 used
GddContext: 91419049984 total in 10908 blocks; 121632 free (7 chunks); 91418928352 used
GddContext: 91435827200 total in 10910 blocks; 121640 free (6 chunks); 91435705560 used
GddContext: 91477770240 total in 10915 blocks; 121688 free (6 chunks); 91477648552 used
GddContext: 91486158848 total in 10916 blocks; 121704 free (6 chunks); 91486037144 used
GddContext: 91494547456 total in 10917 blocks; 121728 free (7 chunks); 91494425728 used
GddContext: 91561656320 total in 10925 blocks; 121800 free (6 chunks); 91561534520 used
GddContext: 91679096832 total in 10939 blocks; 121944 free (6 chunks); 91678974888 used
GddContext: 91687485440 total in 10940 blocks; 121960 free (6 chunks); 91687363480 used
GddContext: 91796537344 total in 10953 blocks; 122112 free (7 chunks); 91796415232 used
GddContext: 91804925952 total in 10954 blocks; 122104 free (6 chunks); 91804803848 used
GddContext: 91813314560 total in 10955 blocks; 122120 free (6 chunks); 91813192440 used
GddContext: 91846868992 total in 10959 blocks; 122176 free (7 chunks); 91846746816 used
GddContext: 91855257600 total in 10960 blocks; 122168 free (6 chunks); 91855135432 used
GddContext: 91863646208 total in 10961 blocks; 122184 free (6 chunks); 91863524024 used
GddContext: 91872034816 total in 10962 blocks; 122208 free (7 chunks); 91871912608 used
GddContext: 91922366464 total in 10968 blocks; 122272 free (7 chunks); 91922244192 used
GddContext: 91930755072 total in 10969 blocks; 122264 free (6 chunks); 91930632808 used
GddContext: 91939143680 total in 10970 blocks; 122280 free (6 chunks); 91939021400 used
GddContext: 91972698112 total in 10974 blocks; 122336 free (7 chunks); 91972575776 used
GddContext: 91981086720 total in 10975 blocks; 122328 free (6 chunks); 91980964392 used
GddContext: 91989475328 total in 10976 blocks; 122344 free (6 chunks); 91989352984 used
GddContext: 91997863936 total in 10977 blocks; 122368 free (7 chunks); 91997741568 used
GddContext: 92056584192 total in 10984 blocks; 122424 free (6 chunks); 92056461768 used
GddContext: 92064972800 total in 10985 blocks; 122440 free (6 chunks); 92064850360 used
GddContext: 92190801920 total in 11000 blocks; 122600 free (6 chunks); 92190679320 used
GddContext: 92266299392 total in 11009 blocks; 122696 free (6 chunks); 92266176696 used
GddContext: 92274688000 total in 11010 blocks; 122720 free (7 chunks); 92274565280 used
GddContext: 92358574080 total in 11020 blocks; 122808 free (6 chunks); 92358451272 used
GddContext: 92366962688 total in 11021 blocks; 122824 free (6 chunks); 92366839864 used
GddContext: 92375351296 total in 11022 blocks; 122848 free (7 chunks); 92375228448 used
GddContext: 92484403200 total in 11035 blocks; 122968 free (6 chunks); 92484280232 used
GddContext: 92492791808 total in 11036 blocks; 122984 free (6 chunks); 92492668824 used
GddContext: 92501180416 total in 11037 blocks; 123008 free (7 chunks); 92501057408 used
GddContext: 92534734848 total in 11041 blocks; 123032 free (6 chunks); 92534611816 used
GddContext: 92559900672 total in 11044 blocks; 123064 free (6 chunks); 92559777608 used
GddContext: 92585066496 total in 11047 blocks; 123096 free (6 chunks); 92584943400 used
GddContext: 92593455104 total in 11048 blocks; 123112 free (6 chunks); 92593331992 used
GddContext: 92635398144 total in 11053 blocks; 123160 free (6 chunks); 92635274984 used
GddContext: 92643786752 total in 11054 blocks; 123176 free (6 chunks); 92643663576 used
GddContext: 92652175360 total in 11055 blocks; 123200 free (7 chunks); 92652052160 used
GddContext: 92660563968 total in 11056 blocks; 123192 free (6 chunks); 92660440776 used
GddContext: 92668952576 total in 11057 blocks; 123208 free (6 chunks); 92668829368 used
GddContext: 92752838656 total in 11067 blocks; 123328 free (7 chunks); 92752715328 used
GddContext: 92761227264 total in 11068 blocks; 123320 free (6 chunks); 92761103944 used
GddContext: 92828336128 total in 11076 blocks; 123424 free (7 chunks); 92828212704 used
GddContext: 92887056384 total in 11083 blocks; 123480 free (6 chunks); 92886932904 used
GddContext: 92895444992 total in 11084 blocks; 123496 free (6 chunks); 92895321496 used
GddContext: 92979331072 total in 11094 blocks; 123616 free (7 chunks); 92979207456 used
GddContext: 92987719680 total in 11095 blocks; 123608 free (6 chunks); 92987596072 used
GddContext: 92996108288 total in 11096 blocks; 123624 free (6 chunks); 92995984664 used
GddContext: 93012885504 total in 11098 blocks; 123640 free (6 chunks); 93012761864 used
GddContext: 93021274112 total in 11099 blocks; 123656 free (6 chunks); 93021150456 used
GddContext: 93088382976 total in 11107 blocks; 123736 free (6 chunks); 93088259240 used
GddContext: 93130326016 total in 11112 blocks; 123808 free (7 chunks); 93130202208 used
GddContext: 93138714624 total in 11113 blocks; 123800 free (6 chunks); 93138590824 used
GddContext: 93147103232 total in 11114 blocks; 123816 free (6 chunks); 93146979416 used
GddContext: 93155491840 total in 11115 blocks; 123840 free (7 chunks); 93155368000 used
GddContext: 93214212096 total in 11122 blocks; 123896 free (6 chunks); 93214088200 used
GddContext: 93222600704 total in 11123 blocks; 123912 free (6 chunks); 93222476792 used
GddContext: 93230989312 total in 11124 blocks; 123936 free (7 chunks); 93230865376 used
GddContext: 93239377920 total in 11125 blocks; 123928 free (6 chunks); 93239253992 used
GddContext: 93272932352 total in 11129 blocks; 123976 free (6 chunks); 93272808376 used
GddContext: 93281320960 total in 11130 blocks; 124000 free (7 chunks); 93281196960 used
GddContext: 93289709568 total in 11131 blocks; 123992 free (6 chunks); 93289585576 used
GddContext: 93381984256 total in 11142 blocks; 124128 free (7 chunks); 93381860128 used
GddContext: 93390372864 total in 11143 blocks; 124120 free (6 ",,,,,,,,"SysLoggerMain","syslogger.c",719,
The memory used by GddContext is growing rapidly
This situation was found twice, and their Final graphs are as follows
### 1、
"global deadlock detected! Final graph is :
{""seg0"":
[""p681 of dtx3113507890 con12475241 waits for a transactionid lock on ShareLock mode, blocked by p686 of dtx3113507891 con12475241"",
""p686 of dtx3113507891 con12475242 waits for a transactionid lock on ShareLock mode, blocked by p661 of dtx3113507880 con12475242"",
""p662 of dtx3113507882 con12475236 waits for a transactionid lock on ShareLock mode, blocked by p686 of dtx3113507891 con12475236"",
""p664 of dtx3113507883 con12475237 waits for a transactionid lock on ShareLock mode, blocked by p676 of dtx3113507888 con12475237"",
""p674 of dtx3113507887 con12475239 waits for a transactionid lock on ShareLock mode, blocked by p664 of dtx3113507883 con12475239""],
""seg1"":
[""p12449 of dtx3113507890 con12475241 waits for a transactionid lock on ShareLock mode, blocked by p12443 of dtx3113507888 con12475241"",
""p12441 of dtx3113507887 con12475239 waits for a transactionid lock on ShareLock mode, blocked by p12449 of dtx3113507890 con12475239"",
""p12443 of dtx3113507888 con12475240 waits for a transactionid lock on ShareLock mode, blocked by p12453 of dtx3113507891 con12475240"",
""p12428 of dtx3113507880 con12475235 waits for a transactionid lock on ShareLock mode, blocked by p12429 of dtx3113507882 con12475235"",
""p12431 of dtx3113507883 con12475237 waits for a transactionid lock on ShareLock mode, blocked by p12449 of dtx3113507890 con12475237""],
""seg2"":
[""p25779 of dtx3113507883 con12475237 waits for a transactionid lock on ShareLock mode, blocked by p25789 of dtx3113507887 con12475237"",
""p25776 of dtx3113507880 con12475235 waits for a transactionid lock on ShareLock mode, blocked by p25789 of dtx3113507887 con12475235"",
""p25791 of dtx3113507888 con12475240 waits for a transactionid lock on ShareLock mode, blocked by p25801 of dtx3113507891 con12475240"",
""p25796 of dtx3113507890 con12475241 waits for a transactionid lock on ShareLock mode, blocked by p25801 of dtx3113507891 con12475241"",
""p25777 of dtx3113507882 con12475236 waits for a transactionid lock on ShareLock mode, blocked by p25801 of dtx3113507891 con12475236"",
""p25789 of dtx3113507887 con12475239 waits for a transactionid lock on ShareLock mode, blocked by p25801 of dtx3113507891 con12475239""]}",
### 2、
global deadlock detected!Final graph is :
{""seg0"":
[""p1241 of dtx3430845286 con1088326 waits for a transactionid lock on ShareLock mode, blocked by p1232 of dtx3430845283 con1088326"",
""p1232 of dtx3430845283 con1088324 waits for a transactionid lock on ShareLock mode, blocked by p1225 of dtx3430845280 con1088324"",
""p1237 of dtx3430845285 con1088325 waits for a transactionid lock on ShareLock mode, blocked by p1222 of dtx3430845278 con1088325"",
""p1223 of dtx3430845279 con1088321 waits for a transactionid lock on ShareLock mode, blocked by p1232 of dtx3430845283 con1088321"",
""p1250 of dtx3430845289 con1088328 waits for a transactionid lock on ShareLock mode, blocked by p1237 of dtx3430845285 con1088328""],
""seg1"":
[""p8451 of dtx3430845279 con1088321 waits for a transactionid lock on ShareLock mode, blocked by p8466 of dtx3430845285 con1088321"",
""p8466 of dtx3430845285 con1088325 waits for a transactionid lock on ShareLock mode, blocked by p8478 of dtx3430845289 con1088325"",
""p8460 of dtx3430845283 con1088324 waits for a transactionid lock on ShareLock mode, blocked by p8450 of dtx3430845278 con1088324"",
""p8478 of dtx3430845289 con1088328 waits for a transactionid lock on ShareLock mode, blocked by p8470 of dtx3430845286 con1088328"",
""p8453 of dtx3430845280 con1088322 waits for a transactionid lock on ShareLock mode, blocked by p8478 of dtx3430845289 con1088322"",
""p8457 of dtx3430845281 con1088323 waits for a transactionid lock on ShareLock mode, blocked by p8450 of dtx3430845278 con1088323""],
""seg2"":
[""p26938 of dtx3430845281 con1088323 waits for a transactionid lock on ShareLock mode, blocked by p26933 of dtx3430845279 con1088323"",
""p26959 of dtx3430845289 con1088328 waits for a transactionid lock on ShareLock mode, blocked by p26952 of dtx3430845286 con1088328"",
""p26952 of dtx3430845286 con1088326 waits for a transactionid lock on ShareLock mode, blocked by p26933 of dtx3430845279 con1088326"",
""p26947 of dtx3430845285 con1088325 waits for a transactionid lock on ShareLock mode, blocked by p26938 of dtx3430845281 con1088325"",
""p26932 of dtx3430845278 con1088320 waits for a transactionid lock on ShareLock mode, blocked by p26941 of dtx3430845283 con1088320""]}
Is it possible that it is caused by multiple wait lock processes on one segment? The Final graphs of the two abnormal situations are relatively complex.
Seems to need a core to have a solid proof. Would you please share the log file of coordinator node with us here?
@yanboer We have another good practice to trace similar OOM issue: using the following gdb script to print call stacks:
handle SIGUSR1 nostop
set pagination off
set logging file gdb.output
set logging on
break AllocSetAllocImpl if context == gddContext
continue
set $loop = 100000 # loop count
while ($loop > 0)
bt -10
set $loop=$loop-1
end
set logging off
quit
When you find the memory usage is increasing, run the script immedicately:
gdb --pid={the pid of gdd process} < gdb_script.in
Then provides the gdb.output to us, thanks.
@yanboer We have another good practice to trace similar OOM issue: using the following gdb script to print call stacks:
handle SIGUSR1 nostop set pagination off set logging file gdb.output set logging on break AllocSetAllocImpl if context == gddContext continue set $loop = 100000 # loop count while ($loop > 0) bt -10 set $loop=$loop-1 end set logging off quitWhen you find the memory usage is increasing, run the script immedicately:
gdb --pid={the pid of gdd process} < gdb_script.inThen provides the gdb.output to us, thanks.
Thank you for your suggestion.
If there is any progress, I will synchronize it.
Would you please share the log file of coordinator node with us here?
Sorry, the coordinator log cannot be provided.
@yanboer We have another good practice to trace similar OOM issue: using the following gdb script to print call stacks:我们还有另一个很好的做法来跟踪类似的 OOM 问题：使用以下 gdb 脚本打印调用堆栈：
handle SIGUSR1 nostop set pagination off set logging file gdb.output set logging on break AllocSetAllocImpl if context == gddContext continue set $loop = 100000 # loop count while ($loop > 0) bt -10 set $loop=$loop-1 end set logging off quitWhen you find the memory usage is increasing, run the script immedicately:当您发现内存使用量增加时，立即运行脚本：
gdb --pid={the pid of gdd process} < gdb_script.inThen provides the gdb.output to us, thanks.然后向我们提供gdb.output，谢谢。
Hello, I executed gdb --pid=50495 < gdb_script.in, but there seems to be some problems.
$ gdb --pid=50495 < gdb_script.in; sleep 1; kill 50495;
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 50495
Reading symbols from /usr/local/greenplum-db/bin/postgres...done.
Reading symbols from /lib64/libz.so.1...Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[New LWP 50496]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libbz2.so.1...Reading symbols from /lib64/libbz2.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libbz2.so.1
Reading symbols from /lib64/libxml2.so.2...Reading symbols from /lib64/libxml2.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libxml2.so.2
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libcurl.so.4...Reading symbols from /lib64/libcurl.so.4...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libcurl.so.4
Reading symbols from /lib64/libzstd.so.1...Reading symbols from /lib64/libzstd.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libzstd.so.1
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/liblzma.so.5...Reading symbols from /lib64/liblzma.so.5...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/liblzma.so.5
Reading symbols from /lib64/libidn.so.11...Reading symbols from /lib64/libidn.so.11...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libidn.so.11
Reading symbols from /lib64/libssh2.so.1...Reading symbols from /lib64/libssh2.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libssh2.so.1
Reading symbols from /lib64/libssl3.so...Reading symbols from /lib64/libssl3.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libssl3.so
Reading symbols from /lib64/libsmime3.so...Reading symbols from /lib64/libsmime3.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libsmime3.so
Reading symbols from /lib64/libnss3.so...Reading symbols from /lib64/libnss3.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss3.so
Reading symbols from /lib64/libnssutil3.so...Reading symbols from /lib64/libnssutil3.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libnssutil3.so
Reading symbols from /lib64/libplds4.so...Reading symbols from /lib64/libplds4.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libplds4.so
Reading symbols from /lib64/libplc4.so...Reading symbols from /lib64/libplc4.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libplc4.so
Reading symbols from /lib64/libnspr4.so...Reading symbols from /lib64/libnspr4.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libnspr4.so
Reading symbols from /lib64/libgssapi_krb5.so.2...Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libk5crypto.so.3...Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libcom_err.so.2...Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/liblber-2.4.so.2...Reading symbols from /lib64/liblber-2.4.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/liblber-2.4.so.2
Reading symbols from /lib64/libldap-2.4.so.2...Reading symbols from /lib64/libldap-2.4.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libldap-2.4.so.2
Reading symbols from /lib64/libssl.so.10...Reading symbols from /lib64/libssl.so.10...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libssl.so.10
Reading symbols from /lib64/libcrypto.so.10...Reading symbols from /lib64/libcrypto.so.10...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypto.so.10
Reading symbols from /lib64/libkrb5support.so.0...Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libsasl2.so.3...Reading symbols from /lib64/libsasl2.so.3...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libsasl2.so.3
Reading symbols from /lib64/libselinux.so.1...Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libpcre.so.1...Reading symbols from /lib64/libpcre.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libpcre.so.1
Reading symbols from /lib64/libfreebl3.so...Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libfreebl3.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
AllocSetAlloc (context=0x27b98e8, size=<optimized out>) at aset.c:1132
1132 aset.c: No such file or directory.
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 cyrus-sasl-lib-2.1.26-24.el7_9.x86_64 glibc-2.17-326.el7_9.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-55.el7_9.x86_64 libcom_err-1.42.9-19.el7.x86_64 libcurl-7.29.0-59.el7_9.1.x86_64 libgcc-4.8.5-44.el7.x86_64 libidn-1.28-4.el7.x86_64 libselinux-2.5-15.el7.x86_64 libssh2-1.8.0-4.el7.x86_64 libxml2-2.9.1-6.el7_9.6.x86_64 libzstd-1.5.5-1.el7.x86_64 nspr-4.34.0-3.1.el7_9.x86_64 nss-3.79.0-5.el7_9.x86_64 nss-softokn-freebl-3.79.0-4.el7_9.x86_64 nss-util-3.79.0-1.el7_9.x86_64 openldap-2.4.44-25.el7_9.x86_64 openssl-libs-1.0.2k-26.el7_9.x86_64 pcre-8.32-17.el7.x86_64 xz-libs-5.2.2-2.el7_9.x86_64 zlib-1.2.7-21.el7_9.x86_64
(gdb) Signal Stop Print Pass to program Description
SIGUSR1 No Yes Yes User defined signal 1
(gdb) (gdb) (gdb) Copying output to gdb.output.
(gdb) Already logging to gdb.output.
(gdb) Function "AllocSetAllocImpl" not defined.
Make breakpoint pending on future shared library load? (y or [n]) [answered N; input not from terminal]
(gdb) Continuing.
^C
Program received signal SIGINT, Interrupt.
GddCtxBreakDeadLock (ctx=ctx@entry=0x27b9a20) at gdddetector.c:158
158 gdddetector.c: No such file or directory.
(gdb) Invalid character '#' in expression.
(gdb) > > >Invalid type combination in ordering comparison.
(gdb) (gdb) Done logging to gdb.output.
(gdb) A debugging session is active.
Inferior 1 [process 50495] will be detached.
Quit anyway? (y or n) [answered Y; input not from terminal]
Detaching from program: /usr/local/greenplum-db/bin/postgres, process 50495
$ cat gdb.output
(gdb) Already logging to gdb.output.
(gdb) Function "AllocSetAllocImpl" not defined.
Make breakpoint pending on future shared library load? (y or [n]) [answered N; input not from terminal]
(gdb) Continuing.
Program received signal SIGINT, Interrupt.
GddCtxBreakDeadLock (ctx=ctx@entry=0x27b9a20) at gdddetector.c:158
158 gdddetector.c: No such file or directory.
(gdb) Invalid character '#' in expression.
(gdb) > > >Invalid type combination in ordering comparison.
(gdb) (gdb)
Seems to need a core to have a solid proof. Would you please share the log file of coordinator node with us here?
Hi this is a coredump file for an exception: https://github.com/yanboer/gdd-coredump/releases/download/coredump/core.49236.gz
# RSS usage by gdd is growing all the time. I was worried that it would take too long to generate the coredump file, causing OOM, so I used kill -19 to pause the gdd process, and then used gcore to generate the coredump file.
kill -19 49236
gcore 49236
gzip core.49236
I checked the coredump file and found that the size occupied by /dev/zero is increasing.
(gdb) info proc mapping
Mapped address spaces:
Start Addr End Addr Size Offset objfile
0x400000 0xf1d000 0xb1d000 0x0 /usr/local/greenplum-db/bin/postgres
0x111c000 0x111d000 0x1000 0xb1c000 /usr/local/greenplum-db/bin/postgres
0x111d000 0x113e000 0x21000 0xb1d000 /usr/local/greenplum-db/bin/postgres
0x7f0129f56000 0x7f0129f57000 0x1000 0xb000 /usr/lib64/libnss_files-2.17.so
0x7f0129f57000 0x7f0129f58000 0x1000 0xc000 /usr/lib64/libnss_files-2.17.so
0x7f0129f5e000 0x7f04991be000 0x36f260000 0x0 /dev/zero (deleted)
0x7f049f902000 0x7f049f903000 0x1000 0x1000 /usr/lib64/libfreebl3.so
0x7f049f903000 0x7f049f904000 0x1000 0x2000 /usr/lib64/libfreebl3.so
0x7f049fb64000 0x7f049fb65000 0x1000 0x60000 /usr/lib64/libpcre.so.1.2.0
0x7f049fb65000 0x7f049fb66000 0x1000 0x61000 /usr/lib64/libpcre.so.1.2.0
0x7f049fd6d000 0x7f049fd6e000 0x1000 0x7000 /usr/lib64/libcrypt-2.17.so
......
Do you have any ideas?
Hello, I executed gdb --pid=50495 < gdb_script.in, but there seems to be some problems.
Oh, sorry for it, the previous one is only for 6x. I have updated it for 7x, please try again:
handle SIGUSR1 nostop
set pagination off
set logging file gdb.output
set logging enabled on
break AllocSetAlloc if context == gddContext
continue
set $loop = 10000
while ($loop > 0)
bt -10
set $loop=$loop-1
end
set logging enabled off
quit
And I will check your core file later.
Hello, I executed gdb --pid=50495 < gdb_script.in, but there seems to be some problems.
Oh, sorry for it, the previous one is only for 6x. I have updated it for 7x, please try again:
handle SIGUSR1 nostop set pagination off set logging file gdb.output set logging enabled on break AllocSetAlloc if context == gddContext continue set $loop = 10000 while ($loop > 0) bt -10 set $loop=$loop-1 end set logging enabled off quitAnd I will check your core file later.
Thanks for the suggestion, here is the output file gdb-output.log
Thanks for the suggestion, here is the output file
I have checked the output, all of them are the same call path:
$ grep "#2" gdb-output.log | sort | uniq -c
3000 #2 0x00000000007a8ee9 in new_tail_cell (list=0x27ba308, list=0x27ba308) at list.c:112
$ grep "#3" gdb-output.log | sort | uniq -c
3000 #3 lappend (list=list@entry=0x27ba308, datum=datum@entry=0x7f004cb99010) at list.c:135
$ grep "#4" gdb-output.log | sort | uniq -c
3000 #4 0x0000000000b16368 in GddCtxBreakDeadLock (ctx=ctx@entry=0x27ba1e0) at gdddetector.c:160
Took a look at the logic in GddCtxBreakDeadLock(), though it doesn't pfree maxvid in each loop (memory leak smells), but MemoryContextReset(gddContext) will be called in the caller function (in doDeadLockCheck()) to free all memories.
So, I think it's more likely to be another bug (e.g. huge memory is consumed in one doDeadLockCheck()) rather than a long-time memory leak. (as your graph: 02:09:50 ~ 02:17:55, memory usage was increasing quickly then an OOM happened)
We have committed more debug logging code recently in: https://github.com/greenplum-db/gpdb/commit/f072ca99081f32165625c80b3f1602c7d2219e97, Is it convenient for you to upgrade to the latest 7x code?
(And I also checked the core file, unlikely I cannot use it without a proper binary program. You have to use gpdb's "pickcore" tool to collect it again, please refer the section 7 in https://greenplum.org/gpsupport-support-utility-for-vmware-greenplum/ )
Thanks.
I have checked the output, all of them are the same call path:
Yes, I adjusted the loop variable in gdb_script.in to 3000 and bt -10 to bt- 20
Took a look at the logic in GddCtxBreakDeadLock(), though it doesn't pfree maxvid in each loop (memory leak smells), but MemoryContextReset(gddContext) will be called in the caller function (in doDeadLockCheck()) to free all memories.
However, breakDeadLock is still executing and has not been completed. It seems that the MemoryContextReset(gddContext) method of doDeadLockCheck will not be executed.
In addition to memory exceptions, the CPU of the gdd process continues to occupy 100% during the exception.So I suspect that the CPU and memory exceptions are caused by palloc(sizeof(*new_tail)); in the while loop of GddCtxBreakDeadLock.
The log interval is 1s
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25759 postgres 20 0 14.0g 47504 44764 S 0.0 0.0 0:00.12 postgres
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25759 postgres 20 0 14.0g 47504 44764 S 0.0 0.0 0:00.12 postgres
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25759 postgres 20 0 14.0g 47504 44764 S 0.0 0.0 0:00.12 postgres
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25759 postgres 20 0 14.7g 753772 47664 R 71.0 0.6 0:00.83 postgres
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25759 postgres 20 0 15.6g 1.7g 47664 R 101.0 1.4 0:01.84 postgres
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25759 postgres 20 0 16.6g 2.6g 47664 R 100.0 2.1 0:02.84 postgres
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25759 postgres 20 0 17.5g 3.6g 47664 R 100.0 2.9 0:03.84 postgres
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25759 postgres 20 0 18.5g 4.5g 47664 R 100.0 3.7 0:04.84 postgres
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25759 postgres 20 0 19.1g 5.1g 47660 t 62.4 4.1 0:05.47 postgres
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25759 postgres 20 0 20.0g 6.0g 47644 R 97.0 4.9 0:06.44 postgres
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25759 postgres 20 0 21.0g 7.0g 47644 R 99.0 5.7 0:07.43 postgres
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25759 postgres 20 0 21.9g 8.0g 47644 R 101.0 6.4 0:08.44 postgres
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25759 postgres 20 0 22.9g 8.9g 47644 R 100.0 7.2 0:09.44 postgres
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25759 postgres 20 0 23.8g 9.9g 47644 R 99.0 8.0 0:10.43 postgres
You have to use gpdb's "pickcore" tool to collect it again, please refer the section 7 in https://greenplum.org/gpsupport-support-utility-for-vmware-greenplum/ )
Thanks for the suggestion, but I don't actually have the gpsupport program.
$ gpsupport --help
-bash: gpsupport: command not found
Is there any way to get the variables that occupy the most memory in the gdd process?
There is a tool called packcore in gpdb, see method 2 in https://community.pivotal.io/s/article/How-to-Collect-Core-Files-for-Analysis?language=en_US
There is a tool called packcore in gpdb, see method 2 in https://community.pivotal.io/s/article/How-to-Collect-Core-Files-for-Analysis?language=en_US
Thanks for the suggestion, here is the packcore file (Because the core file is large and has been uploaded separately before, I commented out shutil.copy(coreFile, '.') of packcore's packCoreFile method.)
packcore：https://github.com/yanboer/gdd-coredump/releases/download/coredump/packcore-core.49236.tgz core：https://github.com/yanboer/gdd-coredump/releases/download/coredump/core.49236.gz