Reconsider number of threads long-polling redis

[olttwa]

Issue

Goose polls redis n times for n threads. To reduce load on redis, we might want to consider polling from just 1 thread, and enqueuing jobs' execution to the threadpool. To limit execution parallelism/concurrency to the user config, we can have an in-memory buffered queue.

N worker instances with T threads polling means O(N*T) operations per second (or long-polling if #65 gets resolved) slamming Redis.

What next?

• Benchmark 2 approaches:
  • Polling redis n times
  • Polling redis once

While benchmarking, measure 2 things:

• time taken to complete 1000 jobs averaging 50ms execution time
• redis memory/CPU consumption