Upgrading `0.2.5` to `0.3.1` hit some "Redlock::LockAcquisitionError" issues

[jeffbax]

I was getting these errors after upgrading:

Redlock::LockAcquisitionError: failed to acquire lock on 'Too many Redis errors prevented lock acquisition: Redis::CommandError: NOSCRIPT No matching script. Please use EVAL.'

Which eventually got be bumbling around to this related issue in the redlock gem: https://github.com/leandromoreira/redlock-rb/issues/124

Even though it is not technically this gem's fault, it might be worth calling out somewhere in the upgrade notes that the config.redlock_servers can no longer receive a Redis.new instance, and now must seem to be a RedisClient.new instance instead.

[jairorjuela]

Hi!! X2

I'm getting the following errors when upgrading to Rails 7.1.2:

[bellef]

@jeffbax Same here but my config.redlock_servers was already set with a RedisClient instance (in an Array):

config.redlock_servers = [
    RedisClient.new(...)
]

[jeffbax]

Spent a good while overhauling Redis/RedisClient pools across our app and no longer seeing this as it's all upgraded now -- the call was inside the house.

@bellef FWIW I am using reconnect_attempts: 1 on the client I pass into redlock, if you haven't seen https://github.com/veeqo/activejob-uniqueness/issues/69#issuecomment-1877140691

[bellef]

@jeffbax I'm curious about what you've done to tackle this as I'm still getting these errors event with the reconnect_attemps: 1.

Redlock::LockAcquisitionError:
failed to acquire lock on 'Too many Redis errors prevented lock acquisition:
(Redlock::LockAcquisitionError) RedisClient::ConnectionError: Broken pipe'

[jeffbax]

@jeffbax I'm curious about what you've done to tackle this as I'm still getting these errors event with the reconnect_attemps: 1.

Redlock::LockAcquisitionError:
failed to acquire lock on 'Too many Redis errors prevented lock acquisition:
(Redlock::LockAcquisitionError) RedisClient::ConnectionError: Broken pipe'

Well YMMV but for us, the main issue was just the amount of things we had accumulated that were attempting to connect to Redis without much oversight.

1. Sidekiq
2. Flipper
3. Redlock
4. ActionCable
5. Rack-Attack
6. Searchkick

Our app was not utilizing connection pools well at all, with a range of 130-370+ connections to redis depending on the time of the day and rate at which our Heroku dynos were rebooting (which were to make everything worse, consistently running out of memory and doing it quite often)

So I overhauled pooling across our App (as well retooled our dyno sizes) and now connections are more reliably 90-150 (out of 400 allowed) At the same time, I went through all the gem upgrades and made sure as many things that supported RedisClient were correctly doing so and sharing a pool where possible.

In addition, I moved what I could to Memcached (with a Redis fallback) - namely Flipper and Rack Attack.

Which is all a long winded way of saying "user error" on just letting too many things open new connections rapidly over time and taking our eyes off the ball. We were hitting max Redis connections particularly at 0-downtime app deploy, and seeing errors like that.

So I would say that should be your first investigation on whether you're actually in the same situation and maxing out supported connections to Redis.

[bellef]

Ok thanks a lot for your feedback 🙏 I will look into it. 👀