Number of CLOSE_WAIT connections when using with nginx

[madhur-srivastava]

We are trying to use this flask plugin (version=0.4.4) to setup a notification server using redis / uwsgi / gevent and nginx(1.4.7). However we are noticing that on client disconnection many of the connections between nginx and the uwsgi flask application are left in CLOSE_WAIT state meaning that application never sends a FIN to nginx and leaves the connection hanging forever. This gets pretty bad eventually filling up the gevent async queue rendering the server unusable. Please see code below

    # spawn a gevent queue to communicate between redis connection and generator
    q = Queue()
    # subscribe to channel and listen for messages.
    # Spawn listener in an asynchronous greenlet
    redis_connection = redis.Redis(connection_pool=redis_pool)
    sub = gevent.spawn(subscribe_and_listen, channel, q, redis_connection, serialize_websocket)

    try:
        while True:
            # wait on queue and yield message to web socket
            message = q.get()
            ws.send(message)
            ack = ws.receive()
            if ack is None:
                # If no ack is received, the connection
                # is broken. Close gracefully on server side
                ws.close()
                break
    except Exception as e:
        logger.error(str(e))
    finally:
        # Kill the greenlet handling the redis subscription
        sub.kill()
        logger.info("Closing client connection")

I looked into the code further and uwsgi's implementation of websocket close and it seems like the close method is a noop. Please advise if we are doing something wrong here and if this can be fixed. Thanks!

[keredson]

we had this problem too, and did a local patch right before self.stream = None, which fixed it for us.

btw calling close() on the socket wasn't enough. from python docs: (https://docs.python.org/2/library/socket.html)

Note close() releases the resource associated with a connection but does not necessarily close the connection immediately. If you want to close the connection in a timely fashion, call shutdown() before close().

$ diff /usr/lib/python2.7/dist-packages/geventwebsocket/websocket.py.orig /usr/lib/python2.7/dist-packages/geventwebsocket/websocket.py
3c3
< from socket import error
---
> from socket import error, SHUT_RDWR
372a373,379
> 
>             try:
>                 # if we don't close, leaks open files in a CLOSE_WAIT state
>                 self.stream.handler.socket.shutdown(SHUT_RDWR)
>                 self.stream.handler.socket.close()
>             except:
>                 pass

note there is a comment in that method:

Close the websocket and connection, sending the specified code and message. The underlying socket object is not closed, that is the responsibility of the initiator.

so whether this is "right" or not is arguable, but EOD it solved it for us.

[zeekay]

Yeah that makes sense. Wanna PR? Happy to merge in your changes :)

[keredson]

i would, but it's more a fix for the lib you're using (gevent's websockets) than your project directly. and as i'm not using flask, i would feel uncomfortable proposing a change i don't know how to test. :) possibly a candidate for your _gevent.py tweaks file tho?

[Ivoz]

(gevent's websockets)

gevent does not implement websockets, I'm not sure where you're getting that from.

[keredson]

https://pypi.python.org/pypi/gevent-websocket/