Throttle concurrency locks timing out, not detecting job end

[aaronjensen]

I haven't had a chance to dig into it yet, but today, our throttled jobs stopped processing. Rather, the concurrency locks each had to time out before allowing another job to go through. Things were processing fine, then there was a heroku-redis upgrade, then, hours later, our dynos restarted and throttled jobs stopped registering their completion. As far as I can tell, there were no code changes from when this worked to when it stopped working.

One possible thing of note is that we also use sidekiq-unique-jobs and this particular job was configured with both:

sidekiq_options queue: :medium, lock: :until_executed, on_conflict: :log, log_duplicate_payload: true
sidekiq_throttle concurrency: { limit: 5 }

I haven't had a chance to dig in and debug—for now I've removed the throttle and the jobs process normally.

Any hints on where to look? What might cause something like this? Could the redis upgrade be related?

Thank you!

[Blokh]

Hello, We're having the same issue - we're using strategy in order to control the concurrency of various jobs. e.g. Sidekiq::Throttled::Registry.add(:feed_listing_worker_concurrency_strategy, concurrency: { limit: 80, ttl: 2.hours.to_i })

When I am fetching the Zrange from the strategy key it shows me that it's full even tho there are no jobs running.

What might be the cause of this issue?

Thanks in advance, Daniel

[ixti]

Heya. Do you get any kind of errors? Like could it be possible that somehow process dies without calling Middlewares?

[newbray]

I have also experienced this issue using sidekiq-unique-jobs (without sidekiq-throttled). I think this is the problem for us: https://github.com/mhenrixon/sidekiq-unique-jobs/issues/486 We have disabled that gem for now

[dchun]

I have the same problem as well. Whenever heroku updates my redis server I get a redis timeout error. I don't use unique jobs so I don't think it's exclusive to unique jobs. Anyone figure this out?

[ixti]

I think it could be related to now removed IPC that was using pub/sub. If you have issues with 1.0.0.alpha - feel free to reopen this issue.