sidekiq-limit_fetch
sidekiq-limit_fetch copied to clipboard
Queue leak
I set a limit on a queue, but it occasionally takes up more.
# sidekiq.yml
:max_retries: 0
production:
:concurrency: 8
:queues:
- default
- fast_task
- slow_task
:limits:
default: 8
fast_task: 8
slow_task: 2
# Gemfile.lock
ruby 2.5.1p57
rails (5.1.7)
sidekiq (5.2.10)
connection_pool (~> 2.2, >= 2.2.2)
rack (~> 2.0)
rack-protection (>= 1.5.0)
redis (~> 4.5, < 4.6.0)
sidekiq-limit_fetch (3.4.0)
sidekiq (>= 4)
I want the "slow task" queue to execute at most 2 tasks simultaneously. It works fine most of the time, but sometimes it runs more tasks at the same time, like even 7 tasks.
Same here, using latest limit_fetch, sidekiq 6 latest, and AWS Elasticache redis 5.0.6. Looking at the queue explain shows that it has "lost" the locks on several Processes so thinks it should be safe to run additional copies (which it should not). It seems to happen more often under high CPU load on the sidekiq servers.
I can't seem to replicate this issue, but if you manage to find it, feel free to send a PR