python-memcached return wrong value after crash??

[samsonle1809]

I'm getting the following issue on trying to run this command on IPython. I see python-memcached return wrong value after crash.

Environment

• MacOS
• Python 3.7.4
• IPython 7.7.0
• python-memcached 1.59

Test scenario: Run IPython and enter the following commands

bash-3.2$ ipython
Python 3.7.4 (default, Jul  9 2019, 18:13:23)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.7.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from memcache import Client as Memcache
   ...: CACHE = Memcache(['127.0.0.1:11211'], socket_timeout=0.5)

In [2]: exit
bash-3.2$ clear
bash-3.2$ ipython
Python 3.7.4 (default, Jul  9 2019, 18:13:23)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.7.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from memcache import Client as Memcache
In [2]: CACHE = Memcache(['127.0.0.1:11211'], socket_timeout=0.5)
In [3]: CACHE.flush_all()
In [4]: for i in range (1, 11):
   ...:     print(CACHE.set('sam-%s' % i, i))
   ...:
True
True
True
True
True
True
True
True
True
True

In [5]: for i in range (1, 11):
   ...:     print('sam-%s = %s' % (i, CACHE.get('sam-%s' % i)))
   ...:
sam-1 = 1
sam-2 = 2
sam-3 = 3
sam-4 = 4
sam-5 = 5
sam-6 = 6
sam-7 = 7
sam-8 = 8
sam-9 = 9
sam-10 = 10

In [6]: for i in range(1, 1000):
   ...:     # Run for a while, press Ctrl+C to raise crash!
   ...:     print(CACHE.set('interrupt-%s' % i, i))
   ...:
True
True
True
True
True
True
^C---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
<ipython-input-6-b57eaab76114> in <module>
      1 for i in range(1, 1000):
      2     # Run for a while, press Ctrl+C to raise crash!
----> 3     print(CACHE.set('interrupt-%s' % i, i))
      4

/usr/local/lib/python3.7/site-packages/python_memcached-1.59-py3.7.egg/memcache.py in set(self, key, val, time, min_compress_len, noreply)
    725         send the reply.
    726         '''
--> 727         return self._set("set", key, val, time, min_compress_len, noreply)
    728
    729     def cas(self, key, val, time=0, min_compress_len=0, noreply=False):

/usr/local/lib/python3.7/site-packages/python_memcached-1.59-py3.7.egg/memcache.py in _set(self, cmd, key, val, time, min_compress_len, noreply)
   1050
   1051         try:
-> 1052             return _unsafe_set()
   1053         except _ConnectionDeadError:
   1054             # retry once

/usr/local/lib/python3.7/site-packages/python_memcached-1.59-py3.7.egg/memcache.py in _unsafe_set()
   1042                 if noreply:
   1043                     return True
-> 1044                 return server.expect(b"STORED", raise_exception=True) == b"STORED"
   1045             except socket.error as msg:
   1046                 if isinstance(msg, tuple):

/usr/local/lib/python3.7/site-packages/python_memcached-1.59-py3.7.egg/memcache.py in expect(self, text, raise_exception)
   1461
   1462     def expect(self, text, raise_exception=False):
-> 1463         line = self.readline(raise_exception)
   1464         if self.debug and line != text:
   1465             if six.PY3:

/usr/local/lib/python3.7/site-packages/python_memcached-1.59-py3.7.egg/memcache.py in readline(self, raise_exception)
   1447             if index >= 0:
   1448                 break
-> 1449             data = recv(4096)
   1450             if not data:
   1451                 # connection close, let's kill it and raise

KeyboardInterrupt:

In [7]: # python-memcached return wrong value!!!
In [8]: for i in range (1, 11):
   ...:     print('sam-%s = %s' % (i, CACHE.get('sam-%s' % i)))
   ...:
sam-1 = None
sam-2 = 1
sam-3 = 2
sam-4 = 3
sam-5 = 4
sam-6 = 5
sam-7 = 6
sam-8 = 7
sam-9 = 8
sam-10 = 9

In [9]: # python-memcached return wrong value!!!
In [10]: for i in range (1, 11):
    ...:     print('sam-%s = %s' % (i, CACHE.get('sam-%s' % i)))
    ...:
sam-1 = 10
sam-2 = 1
sam-3 = 2
sam-4 = 3
sam-5 = 4
sam-6 = 5
sam-7 = 6
sam-8 = 7
sam-9 = 8
sam-10 = 9

I see that after crash python-memcached still keep connection to Memcached but function get() set() have something wrong.

Has anyone run into the same problem? Is this a bug of python-memcached?

[KPassov]

I ran into the same problem, and it's a problem with python-memcached. I made a python script based on your ipython commands that recreates the bug on most runs.

import time

from multiprocessing import Process
from memcache import Client as Memcache


CACHE = Memcache(['127.0.0.1:11211'], socket_timeout=0.5)

# Set some values in the cache
print("Setting!")
for i in range (1, 11):
    CACHE.set('sam-%s' % i, i)
    print('set: sam-%s = %s' % (i,i))

# Get's all the set values to ensure they are saved
print("Getting!")
for i in range (1, 11):
    print('get: sam-%s: %s' % (i,CACHE.get('sam-%s' % i)))


def worker():
  print('Function started')
  for i in range(1000):
      CACHE.set('interrupt-%s' % i, i)
  print('Function finished')

# Start a new process and kill it while it's setting values in the cache
p = Process(target=worker)
p.start()
time.sleep(0.1)
p.terminate()
print("process killed")

# Get the values again
print("Getting!")
for i in range (1, 11):
    print('get: sam-%s: %s' % (i,CACHE.get('sam-%s' % i)))

The output should look like this:

Setting!
set: sam-1 = 1
set: sam-2 = 2
set: sam-3 = 3
set: sam-4 = 4
set: sam-5 = 5
set: sam-6 = 6
set: sam-7 = 7
set: sam-8 = 8
set: sam-9 = 9
set: sam-10 = 10
Getting!
get: sam-1: 1
get: sam-2: 2
get: sam-3: 3
get: sam-4: 4
get: sam-5: 5
get: sam-6: 6
get: sam-7: 7
get: sam-8: 8
get: sam-9: 9
get: sam-10: 10
Function started
process killed
Getting!
get: sam-1: None
get: sam-2: 1
get: sam-3: 2
get: sam-4: 3
get: sam-5: 4
get: sam-6: 5
get: sam-7: 6
get: sam-8: 7
get: sam-9: 8
get: sam-10: 9

It looks like the problem arises when there are 2 or more processes that uses the same Memcache client object, and one of them crashes while setting a value into the cache. After the crash, the other process with the same Memcached object starts returning wrong key/value pairs.

I fixed it by making sure each process initiates it's own Memcached object and made a lookup based on the process id.

_global_caches = {}
def _get_cache():
    # get current process pid
    pid = current_process().pid

    # See if we have a memcache object with this pid
    if pid not in _global_caches:
        _global_caches[pid] = Memcache(['127.0.0.1:11211'])

    return _global_caches[pid]


# Set some values in the cache
print("Setting!")
for i in range (1, 11):
    _get_cache().set('sam-%s' % i, i)
    print('set: sam-%s = %s' % (i,i))

# Get's all the set values to ensure they are saved
print("Getting!")
for i in range (1, 11):
    print('get: sam-%s: %s' % (i,_get_cache().get('sam-%s' % i)))


def worker():
  print('Function started')
  for i in range(1000):
      _get_cache().set('interrupt-%s' % i, i)
  print('Function finished')

# Start a new process that we kill while it's setting values in the 
p = Process(target=worker)
p.start()
time.sleep(0.1)
p.terminate()
print("process killed")

print("Getting!")
for i in range (1, 11):
    print('get: sam-%s: %s' % (i,_get_cache().get('sam-%s' % i)))

This should output the more expected

Setting!
set: sam-1 = 1
set: sam-2 = 2
set: sam-3 = 3
set: sam-4 = 4
set: sam-5 = 5
set: sam-6 = 6
set: sam-7 = 7
set: sam-8 = 8
set: sam-9 = 9
set: sam-10 = 10
Getting!
get: sam-1: 1
get: sam-2: 2
get: sam-3: 3
get: sam-4: 4
get: sam-5: 5
get: sam-6: 6
get: sam-7: 7
get: sam-8: 8
get: sam-9: 9
get: sam-10: 10
Function started
process killed
Getting!
get: sam-1: 1
get: sam-2: 2
get: sam-3: 3
get: sam-4: 4
get: sam-5: 5
get: sam-6: 6
get: sam-7: 7
get: sam-8: 8
get: sam-9: 9
get: sam-10: 10

Just note that this solution uses more sockets for the memcache, as it creates one for each process

[samsonle1809]

Yes, the problem arises when there are more processes that uses the same Memcache client object. I switched to pymemcache library. It has not the same problem.