Blocking on connect to redis blocks the whole process

[gward]

If I make a query to redis that needs to open a connection, and that connection blocks (eg. slow network, network partition), then the whole tornado process blocks. No requests can be handled, even requests that do not depend on redis.

Example: here is a simple tornado HTTP service that handles two URLs, /r1 and /r2. /r1 does not depend on redis at all: it immediately returns a hardcoded string. /r2 fetches a value from redis and returns it.

import tornadoredis
import tornado.httpserver
import tornado.web
import tornado.ioloop
import tornado.gen


class R1Handler(tornado.web.RequestHandler):
    '''handle a request which does not depend on redis'''
    @tornado.web.asynchronous
    @tornado.gen.engine
    def get(self):
        self.write('all is well\n')
        self.finish()


class R2Handler(tornado.web.RequestHandler):
    '''handle a request which does depend on redis'''
    @tornado.web.asynchronous
    @tornado.gen.engine
    def get(self):
        c = tornadoredis.Client()
        foo = yield tornado.gen.Task(c.get, 'foo')
        self.set_header('Content-Type', 'text/plain')
        self.write('foo = %s\n' % (foo,))
        self.finish()


application = tornado.web.Application([
    (r'/r1', R1Handler),
    (r'/r2', R2Handler),
])


if __name__ == '__main__':
    # Start the data initialization routine
    http_server = tornado.httpserver.HTTPServer(application)
    http_server.listen(8888)
    print 'Demo is runing at 0.0.0.0:8888\nQuit the demo with CONTROL-C'
    tornado.ioloop.IOLoop.instance().start()

(I put that script in demos/mixed/app.py in the tornado-redis source tree.)

Expected outcome: if my tornado process is unable to connect to redis, then I expect queries to /r2 to fail or block, but queries to /r1 should continue to work fine.

Actual outcome: as soon as a request for /r2 blocks, requests for /r1 also block. It appears that the entire process is blocked connecting to redis.

Here's how I reproduced it. First, in one terminal run the tornado server:

$ PYTHONPATH=. python demos/mixed/app.py
Demo is runing at 0.0.0.0:8888
Quit the demo with CONTROL-C

In another window, start polling /r1:

$ while true ; do echo -n `date`": "  ; curl http://localhost:8888/r1 ; sleep 1 ; done
Mon Jan 12 13:25:01 EST 2015: all is well
Mon Jan 12 13:25:02 EST 2015: all is well
Mon Jan 12 13:25:03 EST 2015: all is well
[...]

In a third window, ensure that /r2 works:

$ redis-cli 
127.0.0.1:6379> set foo "hello world"
OK
$ curl http://localhost:8888/r2
foo = hello world

Now simulate a network partition: make all packets to redis disappear (this assumes Linux):

$ sudo iptables -F              # wipe all existing firewall rules
$ sudo iptables -A INPUT --proto tcp --dport 6379 -j DROP    # drop packets to redis

Hop over to the window that is polling /r1; it should still be working fine:

[...]
Mon Jan 12 13:27:50 EST 2015: all is well
Mon Jan 12 13:27:51 EST 2015: all is well
Mon Jan 12 13:27:52 EST 2015: all is well
[...]

Now request /r2, which blocks because redis is on the other side of a network partition:

$ curl http://localhost:8888/r2

This request blocks, which is entirely OK. (I'd like it to fail with a timeout eventually, but whatever. Not important.)

However, the /r1 poll is now blocked:

[...]
Mon Jan 12 13:28:28 EST 2015: all is well
Mon Jan 12 13:28:29 EST 2015: all is well
Mon Jan 12 13:28:30 EST 2015: [blocked here]

We can see where it's blocked by hitting Ctrl-C on the tornado process:

^CTraceback (most recent call last):
  File "demos/mixed/app.py", line 40, in <module>
    tornado.ioloop.IOLoop.instance().start()
  File "/usr/lib/python2.7/dist-packages/tornado/ioloop.py", line 607, in start
    self._run_callback(callback)
  File "/usr/lib/python2.7/dist-packages/tornado/ioloop.py", line 458, in _run_callback
    callback()
  File "/usr/lib/python2.7/dist-packages/tornado/stack_context.py", line 331, in wrapped
    raise_exc_info(exc)
  File "/usr/lib/python2.7/dist-packages/tornado/stack_context.py", line 302, in wrapped
    ret = fn(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/tornado/iostream.py", line 341, in wrapper
    callback(*args)
  File "/usr/lib/python2.7/dist-packages/tornado/stack_context.py", line 331, in wrapped
    raise_exc_info(exc)
  File "/usr/lib/python2.7/dist-packages/tornado/stack_context.py", line 302, in wrapped
    ret = fn(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/tornado/httpserver.py", line 327, in _on_headers
    self.request_callback(self._request)
  File "/usr/lib/python2.7/dist-packages/tornado/web.py", line 1600, in __call__
    handler._execute(transforms, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/tornado/web.py", line 1134, in _execute
    self._when_complete(self.prepare(), self._execute_method)
  File "/usr/lib/python2.7/dist-packages/tornado/web.py", line 1141, in _when_complete
    callback()
  File "/usr/lib/python2.7/dist-packages/tornado/web.py", line 1162, in _execute_method
    self._when_complete(method(*self.path_args, **self.path_kwargs),
  File "/usr/lib/python2.7/dist-packages/tornado/web.py", line 1311, in wrapper
    return result
  File "/usr/lib/python2.7/dist-packages/tornado/stack_context.py", line 198, in __exit__
    return self.exception_handler(type, value, traceback)
  File "/usr/lib/python2.7/dist-packages/tornado/web.py", line 1115, in _stack_context_handle_exception
    raise_exc_info((type, value, traceback))
  File "/usr/lib/python2.7/dist-packages/tornado/web.py", line 1298, in wrapper
    result = method(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 159, in wrapper
    deactivate()
  File "/usr/lib/python2.7/dist-packages/tornado/stack_context.py", line 198, in __exit__
    return self.exception_handler(type, value, traceback)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 136, in handle_exception
    return runner.handle_exception(typ, value, tb)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 556, in handle_exception
    self.run()
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 505, in run
    yielded = self.gen.throw(*exc_info)
  File "demos/mixed/app.py", line 23, in get
    foo = yield tornado.gen.Task(c.get, 'foo')
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 153, in wrapper
    runner.run()
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 533, in run
    self.yield_point.start(self)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 371, in start
    self.func(*self.args, **self.kwargs)
  File "/data/src/tornado-redis/tornadoredis/client.py", line 690, in get
    self.execute_command('GET', key, callback=callback)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 159, in wrapper
    deactivate()
  File "/usr/lib/python2.7/dist-packages/tornado/stack_context.py", line 198, in __exit__
    return self.exception_handler(type, value, traceback)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 136, in handle_exception
    return runner.handle_exception(typ, value, tb)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 556, in handle_exception
    self.run()
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 505, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 153, in wrapper
    runner.run()
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 507, in run
    yielded = self.gen.send(next)
  File "/data/src/tornado-redis/tornadoredis/client.py", line 407, in execute_command
    self.connection.connect()
  File "/data/src/tornado-redis/tornadoredis/connection.py", line 72, in connect
    timeout=self.timeout
  File "/usr/lib/python2.7/socket.py", line 562, in create_connection
    sock.connect(sa)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
KeyboardInterrupt

The problem appears to be that tornado-redis is using the blocking socket calls to connect to redis. I suspect it should probably use tornado.tcpclient instead. I am not a tornado expert though! I just spotted that module in the docs.

[leporo]

Sorry for the late reply.

There is a pull request to solve this issue: https://github.com/leporo/tornado-redis/pull/62 It's a shame I haven't checked and integrated it yet.

Could you please check it? There is a chance it can help in your case.

[gward]

Thanks for pointing out that PR. I've reviewed it and tested it: see my comment there.