resec
resec copied to clipboard
Incorrect update of the redis-info to consul
It seems that the redis-info is incorrect updated to consul. The redis info, section 'Replication', on the redis-master server shows two connected slaves:
# Replication
role:master
connected_slaves:2
slave0:ip=10.224.199.37,port=6381,state=online,offset=942807,lag=2
slave1:ip=10.224.199.11,port=6381,state=online,offset=942807,lag=1
master_replid:63e4319d7e22a674e2192e6d1d3fc90e548253fb
master_replid2:5b5e7340f5dd557915f84b8f8b5dfa9d877ca1f6
While the the health check of the same redis-master server in consul shows:
# Replication
role:master
connected_slaves:0
master_replid:63e4319d7e22a674e2192e6d1d3fc90e548253fb
master_replid2:5b5e7340f5dd557915f84b8f8b5dfa9d877ca1f6
Resec was started with these environment vars:
REDIS_ADDR="10.224.199.21:6381"
CONSUL_SERVICE_PREFIX="redis-cache"
CONSUL_LOCK_SESSION_NAME="resec/cache"
CONSUL_LOCK_KEY="resec/cache.lock"
Edit: It seems that pretty much all stats information are affected as well.
Is this intentional, did I miss something or is this a bug?
Hi @MatthiasWiesner! I'm also faced with such a problem during testing version 0.6. I suppose this is happening because of Consul service logic. When ReSec chooses Master it will immediately register it as a service with Redis info output. At that moment Slave servers are still not enabled for replication via ReSec and Master server doesn't know anything about them. After Master being elected Slaves also registering in Consul this time with proper output. But that doesn't trigger Master node to update its info in Consul service. We might possibly ask devs to update service when Redis info output changes
@RedStalker actually it's not a resec bug, but consul config: https://www.consul.io/docs/agent/options.html#check_update_interval so by default what you see in UI is updated every 5 minutes, but you can configure it to whatever time you want!
Hello @burdandrei!
Thank you for your answer! I've created a demo using a docker-compose file in the repo and I was unable to fix the issue from the Consul configuration side. After waining more than 5 minutes, Service check output still showing
connected_slaves:0
Modifing Consul configuration with additional config also doesn't help
{ "check_update_interval": "0s" }
Maybe I'm missing something?
This has been fixed in the 0.6 branch I'm working on over in #30
Hello @jippi ! I'm using source code from branch https://github.com/seatgeek/resec/commits/resec-0.6 and after manual master failover new Master node shows service output when it used to be Slave
Resec: master replication status ServiceName - redis-master CheckID redis@redis-resec-3:6379:replication-status-check Output
Server
redis_version:5.0.6
Replication
role:slave master_host:redis-resec-1 master_port:6379 master_link_status:down
@RedStalker not sure I fully understand your answer, did it fix it for you, or what is your expectation ? :)
@jippi I mean that I'm using code with the latest commits and Service check output in Consul for Master still doesn't show proper information from Redis info.
Steps that I'm doing
- Turn on Consul cluster with 3 nodes
- Start 3 Redis containers and 3 ReSec containers
- Master is being elected in the Consul services
- Slaves are being configured as service in Consul
- Master Service output in Consul shows
connected_slaves:0but the replication is working and Redis-cli inside container showsconnected_slaves:2 - After manually stopping the Master container Slave is being promoted to master and Service output for new Master shows
role:slave master_host:redis-resec-1 master_port:6379 master_link_status:down
It is expected behavior? My expectation is that should the Service update it's output when something changes in Redis replication output?