Running out of Redis sockets, what am I doing wrong?

[sardaukar]

trafficstars

Hello!

Thanks for qup, it looks cool. I've tried using it, but I run out of Redis sockets after a while, so I'm missing some cleanup somewhere.

I have a test repo up here with code exemplifying the problem. Just run bundle, start the worker and the app, and run ./abuse.sh, which calls curl in an endless stream of requests.

1. The requests are published on a topic by the Sinatra app.
2. The worker is subscribed to the topic onto which the Sinatra app dumps the requests, and each requests has a unique id.
3. The worker parses the request and replies on a new "reply" topic, which is derived from the unique id.
4. The Sinatra app picks up the reply, and removes the topic upon success or timeout.

My worker's output:

msgs: 19
msgs: 0
msgs: 33
msgs: 114
msgs: 106
msgs: 114
msgs: 113
msgs: 107
msgs: 110
msgs: 69
msgs: 102
msgs: 104
msgs: 102
msgs: 105
msgs: 102
msgs: 78
msgs: 93
msgs: 76
msgs: 86
msgs: 82
msgs: 79
msgs: 83
msgs: 57
msgs: 96
msgs: 96
msgs: 82
msgs: 102
msgs: 79
msgs: 100
/Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/redis-3.1.0/lib/redis/connection/ruby.rb:13:in `initialize': Too many open files - socket(2) (Errno::EMFILE)
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/redis-3.1.0/lib/redis/connection/ruby.rb:13:in `initialize'
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/redis-3.1.0/lib/redis/connection/ruby.rb:118:in `new'
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/redis-3.1.0/lib/redis/connection/ruby.rb:118:in `connect_addrinfo'
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/redis-3.1.0/lib/redis/connection/ruby.rb:162:in `block in connect'
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/redis-3.1.0/lib/redis/connection/ruby.rb:160:in `each'
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/redis-3.1.0/lib/redis/connection/ruby.rb:160:in `each_with_index'
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/redis-3.1.0/lib/redis/connection/ruby.rb:160:in `connect'
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/redis-3.1.0/lib/redis/connection/ruby.rb:211:in `connect'
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/redis-3.1.0/lib/redis/client.rb:304:in `establish_connection'
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/redis-3.1.0/lib/redis/client.rb:85:in `block in connect'
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/redis-3.1.0/lib/redis/client.rb:266:in `with_reconnect'
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/redis-3.1.0/lib/redis/client.rb:84:in `connect'
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/redis-3.1.0/lib/redis/client.rb:326:in `ensure_connected'
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/redis-3.1.0/lib/redis/client.rb:197:in `block in process'
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/redis-3.1.0/lib/redis/client.rb:279:in `logging'
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/redis-3.1.0/lib/redis/client.rb:196:in `process'
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/redis-3.1.0/lib/redis/client.rb:102:in `call'
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/redis-3.1.0/lib/redis.rb:970:in `block in lpush'
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/redis-3.1.0/lib/redis.rb:37:in `block in synchronize'
    from /Users/sardaukar/.rvm/rubies/ruby-2.1.5/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/redis-3.1.0/lib/redis.rb:37:in `synchronize'
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/redis-3.1.0/lib/redis.rb:969:in `lpush'
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/qup-1.4.1/lib/qup/adapter/redis/queue.rb:71:in `produce'
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/qup-1.4.1/lib/qup/adapter/redis/topic.rb:61:in `block in publish'
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/qup-1.4.1/lib/qup/adapter/redis/topic.rb:60:in `each'
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/qup-1.4.1/lib/qup/adapter/redis/topic.rb:60:in `publish'
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/qup-1.4.1/lib/qup/publisher.rb:27:in `publish'
    from worker.rb:22:in `block (2 levels) in 
'
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/qup-1.4.1/lib/qup/adapter/redis/queue.rb:97:in `yield_message'
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/qup-1.4.1/lib/qup/adapter/redis/queue.rb:86:in `consume'
    from /Users/sardaukar/.rvm/gems/ruby-2.1.5/gems/qup-1.4.1/lib/qup/consumer.rb:27:in `consume'
    from worker.rb:12:in `block in 
'
    from worker.rb:11:in `loop'
    from worker.rb:11:in `
'

Since I'm removing the topic, I have no idea what cleanup I am missing. Thanks for any help!

[copiousfreetime]

Thanks! Glad you enjoy it. I'll take a look this weekend and see if I can see what is going on.

[copiousfreetime]

@sardaukar It looks like you have a found a use-case way that qup+redis doesn't work so well. You are creating a huge number of topics, and the way the current way the redis adapter is implemented is that each topic gets its own connection to redis. In your use-case you are creating a number of topics, one for each request it looks like.

So, at the moment, qup probably won't work for this particular use case. I can change it, and have created issue #20 to address this. I should get to it before the end of the week. If you feel like working on it yourself, I'm happy to accept pull requests.

[sardaukar]

I'll give it a shot. Thanks for having a look!

[copiousfreetime]

Excellent, I look forward to your pull request :smile:

[sardaukar]

I have one open now (https://github.com/copiousfreetime/qup/pull/21). Would appreciate your help with the tests :/