Using Redlock, predixy trying to write in replica

[abhishekaj]

I am using Redlock client of ruby for distributed locking and predixy, redis cluster setup. Here is how I am establishing connection - $redis_lock_manager = Redlock::Client.new(["redis://localhost:7617"]) When I am using lock as - $redis_lock_manager.lock('test', 30*1000), it is trying to write on replica and I am getting the following error - Redis::CommandError: ERR Error running script (call to f_a90bf4be8a6e440e3cc24540eb2f9ff1e16fbcc3): @user_script:2: @user_script: 2: -READONLY You can't write against a read only replica.

This is happening with 50% of requests. All the other commands are working fine. I am not able to replicate when I am running it manually. But it is happening on production. Not sure whether it is a predixy issue or redis cluster issue. Can anyone please help with this?

[fortrue]

paste the predixy conf and info command resp

[abhishekaj]

predixy conf -

Include default.conf

Name PredixyCluster Bind 127.0.0.1:7617

WorkerThreads 15 Log /var/log/predixy/test.cluster.log

ClusterServerPool { MasterReadPriority 50 StaticSlaveReadPriority 50 DynamicSlaveReadPriority 50 RefreshInterval 30s ServerTimeout 500ms ServerFailureLimit 10 ServerRetryTimeout 5s KeepAlive 0 Servers { + pro-cluster.apse1.cache.amazonaws.com:6379 } }

Info -

Version:1.0.5 Name:PredixyCluster Bind:127.0.0.1:7617 RedisMode:proxy SingleThread:false WorkerThreads:15 Uptime:1556085491 UptimeSince:2019-04-24 05:58:11

UsedMemory:508880 MaxMemory:0 MaxRSS:7151616 UsedCpuSys:65.524 UsedCpuUser:51.280

Accept:42 ClientConnections:1 TotalRequests:4807 TotalResponses:4781 TotalRecvClientBytes:147126 TotalSendServerBytes:264074 TotalRecvServerBytes:1378931 TotalSendClientBytes:18626

Server:pro-cluster.apse1.cache.amazonaws.com:6379 Role:slave Group:48e1c26cb73955ac64e9e51935094e3e1d2ab477 DC: CurrentIsFail:0 Connections:15 Connect:15 Requests:945 Responses:915 SendBytes:30453 RecvBytes:334552

Server:10.0.2.133:6379 Role:master Group:48e1c26cb73955ac64e9e51935094e3e1d2ab477 DC: CurrentIsFail:0 Connections:15 Connect:15 Requests:2012 Responses:1982 SendBytes:171413 RecvBytes:357224

Server:10.0.1.28:6379 Role:slave Group:48e1c26cb73955ac64e9e51935094e3e1d2ab477 DC: CurrentIsFail:0 Connections:15 Connect:15 Requests:965 Responses:935 SendBytes:31034 RecvBytes:343229

Server:10.0.1.67:6379 Role:slave Group:48e1c26cb73955ac64e9e51935094e3e1d2ab477 DC: CurrentIsFail:0 Connections:15 Connect:15 Requests:969 Responses:939 SendBytes:31174 RecvBytes:343926

LatencyMonitorName:all <= 100 460 10 0.84% <= 200 450 3 1.09% <= 300 16008 61 6.20% <= 400 16222 48 10.23% <= 500 1302 3 10.48% <= 900 2678 3 10.73% <= 1000 7563 8 11.40% <= 1200 1124246 1039 98.49% <= 1400 15016 12 99.50% <= 1600 1411 1 99.58% <= 3500 3130 1 99.66% <= 10000 9994 1 99.75%

    10000                31880                3 100.00%

T 1031 1230360 1193

LatencyMonitorName:get <= 200 159 1 4.55% <= 300 1388 5 27.27% <= 400 3378 10 72.73%

     1000                 6887                6 100.00%

T 536 11812 22

LatencyMonitorName:set

     1000                10362                9 100.00%

T 1151 10362 9

LatencyMonitorName:blist

[fortrue]

I guess the host pro-cluster.apse1.cache.amazonaws.com will be resolved to different ip, so provide an ip:port in your predix conf

[abhishekaj]

@fortrue this worked! Replaced host name with ip:port. But this is not right. During failover ip might change . Also why isn't it working only for that particular lock command?

[fortrue]

Predixy can handle failover, don't use dns