Redis 6 `OpenSSL::SSL::SSLError` when hitting max connections

[verajohne]

OpenSSL::SSL::SSLError is raised when hitting maxclients.

Reproduction:

In this experiment maxclient is set to 40.

50.times do |n|
  puts "Connection #{n}, #{Time.now.utc}"
  Redis.new(url: url, driver: :ruby, ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE }).ping
end

Truncated output:

...
Connection 39, 2021-02-22 21:27:32 UTC
Traceback (most recent call last):
       16: from /Users/vwoolard/.rvm/rubies/ruby-2.7.2/lib/ruby/2.7.0/monitor.rb:202:in `mon_synchronize'
       15: from /Users/vwoolard/.rvm/rubies/ruby-2.7.2/lib/ruby/2.7.0/monitor.rb:202:in `synchronize'
       14: from /Users/vwoolard/.rvm/gems/ruby-2.7.2/gems/redis-4.2.5/lib/redis.rb:69:in `block in synchronize'
       13: from /Users/vwoolard/.rvm/gems/ruby-2.7.2/gems/redis-4.2.5/lib/redis.rb:172:in `block in ping'
       12: from /Users/vwoolard/.rvm/gems/ruby-2.7.2/gems/redis-4.2.5/lib/redis/client.rb:131:in `call'
       11: from /Users/vwoolard/.rvm/gems/ruby-2.7.2/gems/redis-4.2.5/lib/redis/client.rb:237:in `process'
       10: from /Users/vwoolard/.rvm/gems/ruby-2.7.2/gems/redis-4.2.5/lib/redis/client.rb:325:in `logging'
        9: from /Users/vwoolard/.rvm/gems/ruby-2.7.2/gems/redis-4.2.5/lib/redis/client.rb:238:in `block in process'
        8: from /Users/vwoolard/.rvm/gems/ruby-2.7.2/gems/redis-4.2.5/lib/redis/client.rb:386:in `ensure_connected'
        7: from /Users/vwoolard/.rvm/gems/ruby-2.7.2/gems/redis-4.2.5/lib/redis/client.rb:111:in `connect'
        6: from /Users/vwoolard/.rvm/gems/ruby-2.7.2/gems/redis-4.2.5/lib/redis/client.rb:313:in `with_reconnect'
        5: from /Users/vwoolard/.rvm/gems/ruby-2.7.2/gems/redis-4.2.5/lib/redis/client.rb:112:in `block in connect'
        4: from /Users/vwoolard/.rvm/gems/ruby-2.7.2/gems/redis-4.2.5/lib/redis/client.rb:354:in `establish_connection'
        3: from /Users/vwoolard/.rvm/gems/ruby-2.7.2/gems/redis-4.2.5/lib/redis/connection/ruby.rb:302:in `connect'
        2: from /Users/vwoolard/.rvm/gems/ruby-2.7.2/gems/redis-4.2.5/lib/redis/connection/ruby.rb:260:in `connect'
        1: from /Users/vwoolard/.rvm/gems/ruby-2.7.2/gems/redis-4.2.5/lib/redis/connection/ruby.rb:260:in `connect_nonblock'
OpenSSL::SSL::SSLError (_SSL_connect SYSCALL returned=5 errno=0 state=SSLv3/TLS write client hello_)

In Redis 5 we are used to the familiar ERR max number of clients reached. Digging in the Redis codebase we see that because the TLS Handshake has not been done yet, the connection is just dropped rather than raising a connection error (See Redis code for max client handling)

We ran a similar experiment using the redis-py and noticed that it threw a connection error self._sock = sock ConnectionError: Error 8 connecting to host.compute-1.amazonaws.com:20189. EOF occurred in violation of protocol (_ssl.c:1123) ruby-py code

While it is clear that dropping the connection is intended behavior by Redis for TLS, it is confusing to get an SSL error rather than a connection error or a maxclients exceeded error in this scenario. Would it be possible to rescue the SSLError in this case and raise a ConnectionError here instead? We’d be happy to open a pull request.

[denys-chaikovskyi]

Experienced the same error when hitting maxclients (40 connection on Heroku Redis instance). Would be really great to have more explicit version of error similar to ERR max number of clients reached as @verajohne pointed like it used to be in Redis 5.

It took me some time to figure out that SSL_connect SYSCALL returned=5 errno=0 state=SSLv3/TLS write client hello was indicating that maxclients had been reached 😅

I'm also curious, would it be a good solution to rescue this particular error in the Redis client and wrap it around some form of more explicit version of error pointing that connection limit has been reached or is it better to implement logic to handle this situation within the Redis itself?

Thanks for great project Redis team!

[jrochkind]

Just found this googling, thanks for making the issue so I could! I was also scratching my head at SSL_connect SYSCALL returned=5 errno=0 state=SSLv3/TLS write client hello, but after seeing this -- I think it's likely that exceeding maximum connections is what was going on for me too. (We use a the heroku cloud-hosted redis-as-a-service that meters connections depending on how much you are paying, and possibly had too few available for our architecture).

It would be awesome if this were a more explicit error message somehow.

[byroot]

Sounds like a redis-server issue to me. I don't see what we could do on the client side, the server drop the connection without doing the SSL handshake, from redis-rb's point of view this is indistinguishable from legit SSL issues.

I suggest to report this to redis.

[petrsigut]

We encountered the same behaviour on Heroku (with ActionCables) and would be awesome to be able to rescue from that error - @verajohne any luck with reporting this to redis server?😇 Thank you!🙏