veeqo
Adjust to new redlock behaviour
There have been some changes to the behaviour of redis-client and redlock-rb which lead to a different behaviour in this gem.
Before https://github.com/leandromoreira/redlock-rb/pull/110 was merged, redlock would return false if there was an error with the connection, now it raises the error.
Before this behaviour could have been handled by configuring on_conflict. Now the behaviour should be more fine grained.
This adds a on_error option, which can be either nil (leading to a reraise) or a proc leading to the evaluation of said proc.
This allows users to handle connections errors to reduce the amount of jobs that get lost because of said connection errors.
I've been running this PR in production for some days now. I added some logging around that callback, but it appears to be silent. So this might not be an issue at all. Even though this PR might still be a feature worth adding.
Most of the errors that we encounter come from delete_lock. So I will try and investigate that if I find the time, maybe there is something we can do about that.
redlock's strategy is to simply fire and forget the unlock attempt, rescuing every error that might occur. But I don't know about that, since redlock is more about process based locking. Trying the same might lead to a deadlock situation.
Think we've also been encountering this issue after doing an upgrade with a job with the unique config of; unique :until_executed, lock_ttl: 10.minutes, on_conflict: :log.
Is the gem going to include these changes in the near future?
I think we are experiencing a similar issue related to the redis-client and redlock-rb changes. @nduitz are you still running this branch?
@sharshenov are we able to merge or get an official fix?
I think we are experiencing a similar issue related to the
redis-clientandredlock-rbchanges. @nduitz are you still running this branch?@sharshenov are we able to merge or get an official fix?
Hey, yes I am still running the branch.
But as I said before, this PR doesn't change anything on the situation. We still have loads of RedisClient::ConnectionError which are in fact triggered by delete_lock.
Since we never experienced any real issues I suspect this arrises through a raise condition and the lock already being deleted but I did not investigate it further
I apologize for the delay. The change is merged now and it will be released with the next gem version
UPD released https://rubygems.org/gems/activejob-uniqueness/versions/0.4.0
