pftaskqueue icon indicating copy to clipboard operation
pftaskqueue copied to clipboard

Implement bulk key fetching and deleting for large queue

Open shioshiota opened this issue 2 years ago • 0 comments

We want to reduce redis load by delete-queue command when the queue has a large number of tasks. *a large number is about 10^6

I implemented following improvements:

  • use SScan instead of SMembers command to fetch task UIDs
    • https://github.com/pfnet-research/pftaskqueue/pull/44/commits/21b33e1ddf7fa3882d14a6340a84613726b733cd
  • use UnLink instead of Del command and bulk deleting keys by chunk size
    • https://redis.io/commands/UNLINK
    • https://github.com/pfnet-research/pftaskqueue/pull/44/commits/2b99cf7d427f0a8feab8066b7d93c286163f5378

fixes https://github.com/pfnet-research/pftaskqueue/issues/28

shioshiota avatar Sep 07 '21 05:09 shioshiota