Eject Down Servers

[eminugurkenar]

Version: 35.0.0-master mcrouter

Hi, I am testing FailoverRoute with configuration below.

The test

1-) set a key 'echo -e 'set mykey 0 600 5\r\nhello1\r' | nc localhost 11311'
2-) block connection to port 11211 on server that 'mykey' goes. 'iptables -A INPUT -p tcp --dport 11211 -j DROP' 
3-) delete key 'echo -e 'delete mykey\r' | nc localhost 11311'
4-) set same key again 'echo -e 'set mykey 0 600 5\r\nhello2\r' | nc localhost 11311' (key goes to failover server)
5-) get key 'echo -e 'mykey a\r' | nc localhost 11311' (returns hello2)
6-) flush iptables rules on main server. iptables -F
7-) get key 'echo -e 'mykey a\r' | nc localhost 11311' (returns hello1)

You can see that if main pool server comebacks , mcrouter returns stale data. As i understand from that issue (https://github.com/facebook/mcrouter/issues/105 jmswen comment), it is our responsibility to apply delete stream to right memcached instance. Right now we do not have any agent that keeps track of the delete stream then applies them to right server.

Is there a way to eject down server from the pool? It will be very usefull for that situation. We can easly flush ejected server then add it back to the pool. That way the problem described above will not be a issue for us.

Configuration:

{
   "pools": {
     "main": {
       "servers": [
         "172.31.84.43:11211",
         "172.31.84.48:11211"
       ]
     },
     "backup": {
       "servers": [
         "172.31.84.201:11211"
       ]
     }
   },
   "route":{
     "type": "FailoverRoute",
     "children" : ["PoolRoute|main", "PoolRoute|backup" ]
   }
}