turbinia icon indicating copy to clipboard operation
turbinia copied to clipboard

Add support for multi-pools for workers

Open aarontp opened this issue 7 years ago • 2 comments

We can add different worker pools that are created to match individual job types (e.g. so Plaso can have worker nodes that have more cpu than workers for other job types). Right now it's one global worker pool.

aarontp avatar May 23 '17 21:05 aarontp

From a GKE autoscaling perspective, we can consider:

  • 1 Large Task gets a single node for processing
  • Multiple small tasks can run multiple pods on a single node

Also handy resource: https://medium.com/google-cloud/scale-your-kubernetes-cluster-to-almost-zero-with-gke-autoscaler-9c78051cbf40

wajihyassine avatar Nov 23 '22 18:11 wajihyassine

Tagging 'celery' because I think we can use built-in functionality for this

jleaniz avatar Jul 21 '23 15:07 jleaniz