pftaskqueue
pftaskqueue copied to clipboard
Implement bulk key fetching and deleting for large queue
We want to reduce redis load by delete-queue command when the queue has a large number of tasks. *a large number is about 10^6
I implemented following improvements:
- use SScan instead of SMembers command to fetch task UIDs
- https://github.com/pfnet-research/pftaskqueue/pull/44/commits/21b33e1ddf7fa3882d14a6340a84613726b733cd
- use UnLink instead of Del command and bulk deleting keys by chunk size
- https://redis.io/commands/UNLINK
- https://github.com/pfnet-research/pftaskqueue/pull/44/commits/2b99cf7d427f0a8feab8066b7d93c286163f5378
fixes https://github.com/pfnet-research/pftaskqueue/issues/28