resec icon indicating copy to clipboard operation
resec copied to clipboard

Incorrect update of the redis-info to consul

Open MatthiasWiesner opened this issue 6 years ago • 7 comments

It seems that the redis-info is incorrect updated to consul. The redis info, section 'Replication', on the redis-master server shows two connected slaves:

# Replication
role:master
connected_slaves:2
slave0:ip=10.224.199.37,port=6381,state=online,offset=942807,lag=2
slave1:ip=10.224.199.11,port=6381,state=online,offset=942807,lag=1
master_replid:63e4319d7e22a674e2192e6d1d3fc90e548253fb
master_replid2:5b5e7340f5dd557915f84b8f8b5dfa9d877ca1f6

While the the health check of the same redis-master server in consul shows:

# Replication
role:master
connected_slaves:0
master_replid:63e4319d7e22a674e2192e6d1d3fc90e548253fb
master_replid2:5b5e7340f5dd557915f84b8f8b5dfa9d877ca1f6

Resec was started with these environment vars:

REDIS_ADDR="10.224.199.21:6381"
CONSUL_SERVICE_PREFIX="redis-cache"
CONSUL_LOCK_SESSION_NAME="resec/cache"
CONSUL_LOCK_KEY="resec/cache.lock"

Edit: It seems that pretty much all stats information are affected as well.

Is this intentional, did I miss something or is this a bug?

MatthiasWiesner avatar May 14 '19 13:05 MatthiasWiesner

Hi @MatthiasWiesner! I'm also faced with such a problem during testing version 0.6. I suppose this is happening because of Consul service logic. When ReSec chooses Master it will immediately register it as a service with Redis info output. At that moment Slave servers are still not enabled for replication via ReSec and Master server doesn't know anything about them. After Master being elected Slaves also registering in Consul this time with proper output. But that doesn't trigger Master node to update its info in Consul service. We might possibly ask devs to update service when Redis info output changes

RedStalker avatar Nov 11 '19 05:11 RedStalker

@RedStalker actually it's not a resec bug, but consul config: https://www.consul.io/docs/agent/options.html#check_update_interval so by default what you see in UI is updated every 5 minutes, but you can configure it to whatever time you want!

burdandrei avatar Nov 11 '19 08:11 burdandrei

Hello @burdandrei! Thank you for your answer! I've created a demo using a docker-compose file in the repo and I was unable to fix the issue from the Consul configuration side. After waining more than 5 minutes, Service check output still showing connected_slaves:0

Modifing Consul configuration with additional config also doesn't help { "check_update_interval": "0s" } Maybe I'm missing something?

RedStalker avatar Nov 11 '19 21:11 RedStalker

This has been fixed in the 0.6 branch I'm working on over in #30

jippi avatar Nov 12 '19 08:11 jippi

Hello @jippi ! I'm using source code from branch https://github.com/seatgeek/resec/commits/resec-0.6 and after manual master failover new Master node shows service output when it used to be Slave

Resec: master replication status ServiceName - redis-master CheckID redis@redis-resec-3:6379:replication-status-check Output

Server

redis_version:5.0.6

Replication

role:slave master_host:redis-resec-1 master_port:6379 master_link_status:down

RedStalker avatar Nov 12 '19 16:11 RedStalker

@RedStalker not sure I fully understand your answer, did it fix it for you, or what is your expectation ? :)

jippi avatar Nov 12 '19 16:11 jippi

@jippi I mean that I'm using code with the latest commits and Service check output in Consul for Master still doesn't show proper information from Redis info.

Steps that I'm doing

  1. Turn on Consul cluster with 3 nodes
  2. Start 3 Redis containers and 3 ReSec containers
  3. Master is being elected in the Consul services
  4. Slaves are being configured as service in Consul
  5. Master Service output in Consul shows connected_slaves:0 but the replication is working and Redis-cli inside container shows connected_slaves:2
  6. After manually stopping the Master container Slave is being promoted to master and Service output for new Master shows

role:slave master_host:redis-resec-1 master_port:6379 master_link_status:down

It is expected behavior? My expectation is that should the Service update it's output when something changes in Redis replication output?

RedStalker avatar Nov 12 '19 16:11 RedStalker