python-task-queue
python-task-queue copied to clipboard
seung-lab
Asynchronous serverless task queue with timed leasing of tasks. Threaded implementations for SQS and local filesystem.
Sometimes particular tasks are extremely memory hungry out of proportion with others and need special attention. We should have a "stasis" flag that causes these tasks to be bypassed until...
Some people are reporting problems with parallel operation hanging. We'll have to figure out how to reproduce.
If we moved completed items into a "completed" directory, we could get a precise count of completed tasks instead of a count of executed tasks.
PR adds support for Google's Pub/Sub queue TODO: Add tests and maybe look into the missing features
It would be useful to support task chaining. This allows processors to guarantee locality when acquiring the next task. The simplest version of this would be to insert a list...
SQS doesn't care about buckets, so even creating secrets with these names does not help.
sqs can return empty messages if the message number is low (
