predixy
predixy copied to clipboard
joyieldInc
Predixy making more connections then desired and breaking connection with redis cluster
We have dedicated 15 worker threads but predixy is making more connections as seen in connect and ultimately breaking connection with redis and establishing new connections. Due to this we are getting high errors on production.
This is predixy conf file ->
Include default.conf Name PredixyCartheroCluster Bind 127.0.0.1:7617
WorkerThreads 15 Log /var/log/predixy/carthero.cluster.log ClusterServerPool { MasterReadPriority 10 StaticSlaveReadPriority 50 DynamicSlaveReadPriority 50 RefreshInterval 30s ServerTimeout 500ms ServerFailureLimit 10 ServerRetryTimeout 5s KeepAlive 0 Servers { + 10.0.2.133:6379 } }
This is info ->
Server:10.0.2.133:6379 Role:master Group:48e1c26cb73955ac64e9e51935094e3e1d2ab477 DC: CurrentIsFail:0 Connections:15 Connect:17 Requests:156910 Responses:156906 SendBytes:41020537 RecvBytes:820555
Error on predixy logs ->
2019-05-30 06:43:07.964114 N Handler.cpp:276 h 1 close s 10.0.2.133:6379 32 and c None -1 with status 102 Custom 2019-05-30 06:43:08.010332 N Handler.cpp:276 h 11 close s 10.0.2.133:6379 76 and c None -1 with status 102 Custom 2019-05-30 06:43:09.781291 N ConnectConnectionPool.cpp:50 h 11 reopen server connection 10.0.2.133:6379 32 2019-05-30 06:43:10.426497 N ConnectConnectionPool.cpp:50 h 1 reopen server connection 10.0.2.133:6379 76
This is happening in multiple servers with master node as well as replica nodes.
This is particularly happening with the 2nd shard of redis cluster.
One more example ->
Predixy info ->
Proxy
Version:1.0.5 Name:PredixyCartheroCluster Bind:127.0.0.1:7617 RedisMode:proxy SingleThread:false WorkerThreads:15 Uptime:1559196691 UptimeSince:2019-05-30 06:11:31
SystemResource
UsedMemory:3971296 MaxMemory:0 MaxRSS:27136000 UsedCpuSys:61.003 UsedCpuUser:33.239
Stats
Accept:284 ClientConnections:16 TotalRequests:1032164 TotalResponses:1032144 TotalRecvClientBytes:106498533 TotalSendServerBytes:106964199 TotalRecvServerBytes:76613370 TotalSendClientBytes:76189355
Servers
Server:10.0.2.133:6379 Role:master Group:48e1c26cb73955ac64e9e51935094e3e1d2ab477 DC: CurrentIsFail:0 Connections:15 Connect:15 Requests:90602 Responses:90585 SendBytes:30099293 RecvBytes:482764
Server:10.0.2.179:6379 Role:slave Group:5897c6250a82009ae15ed2c144f921d61f09c6cb DC: CurrentIsFail:0 Connections:15 Connect:16 Requests:217746 Responses:217729 SendBytes:13251890 RecvBytes:18031740
Server:10.0.1.28:6379 Role:slave Group:48e1c26cb73955ac64e9e51935094e3e1d2ab477 DC: CurrentIsFail:0 Connections:15 Connect:15 Requests:186567 Responses:186550 SendBytes:11608338 RecvBytes:19397866
Server:10.0.2.92:6379 Role:slave Group:5897c6250a82009ae15ed2c144f921d61f09c6cb DC: CurrentIsFail:0 Connections:15 Connect:15 Requests:217047 Responses:217030 SendBytes:13202903 RecvBytes:18527394
Server:10.0.1.67:6379 Role:slave Group:48e1c26cb73955ac64e9e51935094e3e1d2ab477 DC: CurrentIsFail:0 Connections:15 Connect:15 Requests:186355 Responses:186338 SendBytes:11600419 RecvBytes:19500493
Server:10.0.1.187:6379 Role:master Group:5897c6250a82009ae15ed2c144f921d61f09c6cb DC: CurrentIsFail:0 Connections:15 Connect:16 Requests:133930 Responses:133911 SendBytes:27201356 RecvBytes:673113
LatencyMonitor
LatencyMonitorName:all <= 100 122576 1264 0.12% <= 200 49164991 376427 36.61% <= 300 26965194 104790 46.77% <= 400 10665069 33258 50.00% <= 500 243415 556 50.05% <= 600 128516 233 50.07% <= 700 127508 196 50.09% <= 800 156171 208 50.11% <= 900 103452373 115937 61.35% <= 1000 230552904 250865 85.67% <= 1200 111492139 104556 95.80% <= 1400 3432330 2664 96.06% <= 1600 2692813 1800 96.24% <= 1700 1286718 780 96.31% <= 1800 1325369 757 96.39% <= 2000 3098603 1631 96.54% <= 2500 8630715 3841 96.92% <= 3000 10059436 3660 97.27% <= 3500 12259156 3772 97.64% <= 4000 13543698 3611 97.99% <= 4500 16172121 3805 98.36% <= 5000 17741361 3735 98.72% <= 6000 45171595 8203 99.51% <= 7000 29778721 4729 99.97% <= 8000 1517807 211 99.99% <= 9000 170320 20 99.99% <= 10000 95145 10 99.99%
10000 3329548 55 100.00%
T 681 703376312 1031574
LatencyMonitorName:get <= 100 54193 548 0.07% <= 200 41935005 322994 44.01% <= 300 9944499 38244 49.21% <= 400 5456472 16915 51.51% <= 500 112366 259 51.55% <= 600 41924 76 51.56% <= 700 32938 51 51.57% <= 800 27313 36 51.57% <= 900 96788088 108484 66.33% <= 1000 176029677 191938 92.44%
1000 157621556 55595 100.00%
T 663 488044031 735140
LatencyMonitorName:set <= 100 48850 516 0.43% <= 200 800173 5808 5.26% <= 300 10190350 39870 38.41% <= 400 2708791 8554 45.52% <= 500 58296 130 45.63% <= 600 50663 92 45.70% <= 700 52205 80 45.77% <= 800 85564 114 45.86% <= 900 2388133 2674 48.09% <= 1000 18674394 20153 64.84%
1000 58354597 42287 100.00%
T 776 93412016 120278
LatencyMonitorName:blist
Error Logs
2019-05-30 06:11:41.319340 N ConnectConnectionPool.cpp:50 h 1 reopen server connection 10.0.1.187:6379 33 2019-05-30 09:19:58.495717 N ConnectConnectionPool.cpp:50 h 13 reopen server connection 10.0.2.179:6379 42
Please help .
Check the redis slowlog, with status 102 Custom means redis response timeout, your ServerTimeout is 500ms
I have the same issue: Predixy is making extra connections in addition to its persistent connections when I perform many commands and heavy load.
Is there any way to limit these extra connections so that my redis cluster shards won't get overwhelmed?
