pika主库内存突然飙升

[15611118611]

pika进程最近使用内存频繁出现突然飙升，直到被系统Kill，确认连接数很低，且稳定，在10左右。每次出现异常，pika的日志中均有主从断开连接，开始同步的信息。

pmap输出： Address Kbytes Mode Offset Device Mapping 0000000000400000 7176 r-x-- 0000000000000000 0fd:00010 pika 0000000000d01000 44 rw--- 0000000000701000 0fd:00010 pika 0000000000d0c000 464 rw--- 0000000000000000 000:00000 [ anon ] 00000000017bd000 5220 rw--- 0000000000000000 000:00000 [ anon ] 0000000001cd6000 89349816 rw--- 0000000000000000 000:00000 [ anon ] 00007f0327457000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0327458000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0327c58000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0327c59000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0328459000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f032845a000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f032945b000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f032945c000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f032e75b000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f032e75c000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f032ef5c000 4096 rw-s- 0000000005000000 0fd:00010 write2file497347 00007f032f35c000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f032f35d000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f032fb5d000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f032fb5e000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f033035e000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f033035f000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0330b5f000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0330b60000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0331360000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0331361000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0331b61000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0331b62000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0332362000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0332363000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0332b63000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0332b64000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0333364000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0333365000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0333b65000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0333b66000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0334366000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0334367000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0334b67000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0334b68000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0335368000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0335369000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0335b69000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0335b6a000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f033636a000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f033636b000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0336b6b000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0336b6c000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f033736c000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f033736d000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0337b6d000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0337b6e000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f033836e000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f033836f000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0338b6f000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0338b70000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0339370000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0339371000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0339b71000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0339b72000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f033a372000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f033a373000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f033ab73000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f033ab74000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f033b374000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f033b375000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f033bb75000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f033bb76000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f033c376000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f033c377000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f033cb77000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f033cb78000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f033d378000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f033d379000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f033db79000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f033db7a000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f033e37a000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f033e37b000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f033eb7b000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f033eb7c000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f033f37c000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f033f37d000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f033fb7d000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f033fb7e000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f034037e000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f034037f000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0340b7f000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0340b80000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0341380000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0341381000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0341b81000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0341b82000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0342382000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0342383000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0342b83000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0342b84000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0343384000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0343385000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0343b85000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0343b86000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0344386000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0344387000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0344b87000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0344b88000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0345388000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0345389000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0345b89000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0345b8a000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f034638a000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f034638b000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0346b8b000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0346b8c000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f034738c000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f034738d000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0347b8d000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0347b8e000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f034838e000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f034838f000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0348b8f000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0348b90000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0349390000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0349391000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f0349b91000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f0349b92000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f034a392000 4 ----- 0000000000000000 000:00000 [ anon ] 00007f034a393000 8192 rw--- 0000000000000000 000:00000 [ anon ] 00007f034ab93000 1756 r-x-- 0000000000000000 0fd:00001 libc-2.17.so 00007f034ad4a000 2044 ----- 00000000001b7000 0fd:00001 libc-2.17.so 00007f034af49000 16 r---- 00000000001b6000 0fd:00001 libc-2.17.so 00007f034af4d000 8 rw--- 00000000001ba000 0fd:00001 libc-2.17.so 00007f034af4f000 20 rw--- 0000000000000000 000:00000 [ anon ] 00007f034af54000 84 r-x-- 0000000000000000 0fd:00001 libgcc_s-4.8.5-20150702.so.1 00007f034af69000 2044 ----- 0000000000015000 0fd:00001 libgcc_s-4.8.5-20150702.so.1 00007f034b168000 4 r---- 0000000000014000 0fd:00001 libgcc_s-4.8.5-20150702.so.1 00007f034b169000 4 rw--- 0000000000015000 0fd:00001 libgcc_s-4.8.5-20150702.so.1 00007f034b16a000 1024 r-x-- 0000000000000000 0fd:00001 libm-2.17.so 00007f034b26a000 2048 ----- 0000000000100000 0fd:00001 libm-2.17.so 00007f034b46a000 4 r---- 0000000000100000 0fd:00001 libm-2.17.so 00007f034b46b000 4 rw--- 0000000000101000 0fd:00001 libm-2.17.so 00007f034b46c000 60 r-x-- 0000000000000000 0fd:00001 libbz2.so.1.0.6 00007f034b47b000 2044 ----- 000000000000f000 0fd:00001 libbz2.so.1.0.6 00007f034b67a000 4 r---- 000000000000e000 0fd:00001 libbz2.so.1.0.6 00007f034b67b000 4 rw--- 000000000000f000 0fd:00001 libbz2.so.1.0.6 00007f034b67c000 84 r-x-- 0000000000000000 0fd:00001 libz.so.1.2.7 00007f034b691000 2044 ----- 0000000000015000 0fd:00001 libz.so.1.2.7 00007f034b890000 4 r---- 0000000000014000 0fd:00001 libz.so.1.2.7 00007f034b891000 4 rw--- 0000000000015000 0fd:00001 libz.so.1.2.7 00007f034b892000 28 r-x-- 0000000000000000 0fd:00001 librt-2.17.so 00007f034b899000 2044 ----- 0000000000007000 0fd:00001 librt-2.17.so 00007f034ba98000 4 r---- 0000000000006000 0fd:00001 librt-2.17.so 00007f034ba99000 4 rw--- 0000000000007000 0fd:00001 librt-2.17.so 00007f034ba9a000 92 r-x-- 0000000000000000 0fd:00001 libpthread-2.17.so 00007f034bab1000 2044 ----- 0000000000017000 0fd:00001 libpthread-2.17.so 00007f034bcb0000 4 r---- 0000000000016000 0fd:00001 libpthread-2.17.so 00007f034bcb1000 4 rw--- 0000000000017000 0fd:00001 libpthread-2.17.so 00007f034bcb2000 16 rw--- 0000000000000000 000:00000 [ anon ] 00007f034bcb6000 128 r-x-- 0000000000000000 0fd:00001 ld-2.17.so 00007f034beb1000 8 rw--- 0000000000000000 000:00000 [ anon ] 00007f034beb3000 64 rw-s- 0000000000000000 0fd:00010 manifest 00007f034bec3000 4 rw--- 0000000000000000 000:00000 [ anon ] 00007f034bec4000 32 rw--- 0000000000000000 000:00000 [ anon ] 00007f034becc000 20 rw--- 0000000000000000 000:00000 [ anon ] 00007f034bed1000 16 rw--- 0000000000000000 000:00000 [ anon ] 00007f034bed5000 4 r---- 000000000001f000 0fd:00001 ld-2.17.so 00007f034bed6000 4 rw--- 0000000000020000 0fd:00001 ld-2.17.so 00007f034bed7000 4 rw--- 0000000000000000 000:00000 [ anon ] 00007ffcfad51000 3088 rw--- 0000000000000000 000:00000 [ stack ] 00007ffcfb170000 8 r-x-- 0000000000000000 000:00000 [ anon ] ffffffffff600000 4 r-x-- 0000000000000000 000:00000 [ anon ] mapped: 89879508K writeable/private: 89850308K shared: 4160K

主库相关日志信息：

W0620 08:56:18.437028 30976 pika_rm.cc:530] (db0:0) Master del Recv Timeout slave success 10.1.1.1:9220

I0620 08:58:15.101115 30905 pika_repl_server_thread.cc:29] ServerThread Close Slave Conn, fd: 830, ip_port: 10.1.1.1:33638 I0620 08:58:15.101493 30905 pika_server.cc:646] Delete Slave Success, ip_port: 10.1.1.1:9220 I0620 08:58:15.186741 30907 pika_server.cc:742] Add New Slave, 10.1.1.1:9220 I0620 08:58:15.286247 30907 pika_repl_server_conn.cc:107] Receive Trysync, Slave ip: 10.1.1.1, Slave port:9220, Partition: db0, filenum: 497069, pro_offset: 84090084 I0620 08:58:15.286489 30907 pika_repl_server_conn.cc:137] Partition: db0 binlog has been purged, may need full sync I0620 08:58:15.386689 30908 pika_repl_server_conn.cc:230] Handle partition DBSync Request I0620 08:58:15.387486 30908 pika_rm.cc:163] Add Slave Node, partition: (db0:0), ip_port: 10.1.1.1:9220 I0620 08:58:15.388350 30908 pika_repl_server_conn.cc:245] Partition: db0_0 Handle DBSync Request Success, Session: 2 I0620 08:58:16.370102 31009 pika_partition.cc:424] db0 after prepare bgsave I0620 08:58:16.376474 31009 pika_partition.cc:427] db0 bgsave_info: path=/home/deploy/pika/pika_9220/dump/20210620/db0, filenum=497081, offset=88113948 I0620 08:58:17.214248 31009 pika_partition.cc:433] db0 create new backup finished. I0620 08:58:17.214531 31009 pika_server.cc:936] Partition: db0 Start Send files in /home/deploy/pika/pika_9220/dump/20210620/db0 to 10.1.1.1