sidekiq-limit_fetch icon indicating copy to clipboard operation
sidekiq-limit_fetch copied to clipboard

Queue leak

Open Skyrain-Young opened this issue 2 years ago • 2 comments

I set a limit on a queue, but it occasionally takes up more.

# sidekiq.yml
:max_retries: 0
production:
  :concurrency: 8
:queues:
  - default
  - fast_task
  - slow_task
:limits:
  default: 8
  fast_task: 8
  slow_task: 2
# Gemfile.lock
ruby 2.5.1p57
rails (5.1.7)
sidekiq (5.2.10)
  connection_pool (~> 2.2, >= 2.2.2)
  rack (~> 2.0)
  rack-protection (>= 1.5.0)
  redis (~> 4.5, < 4.6.0)
sidekiq-limit_fetch (3.4.0)
  sidekiq (>= 4)

I want the "slow task" queue to execute at most 2 tasks simultaneously. It works fine most of the time, but sometimes it runs more tasks at the same time, like even 7 tasks.

Skyrain-Young avatar Sep 13 '23 02:09 Skyrain-Young

Same here, using latest limit_fetch, sidekiq 6 latest, and AWS Elasticache redis 5.0.6. Looking at the queue explain shows that it has "lost" the locks on several Processes so thinks it should be safe to run additional copies (which it should not). It seems to happen more often under high CPU load on the sidekiq servers.

matt-domsch-sp avatar Feb 25 '24 21:02 matt-domsch-sp

I can't seem to replicate this issue, but if you manage to find it, feel free to send a PR

deanpcmad avatar Feb 26 '24 17:02 deanpcmad