turbinia
turbinia copied to clipboard
Add support for multi-pools for workers
We can add different worker pools that are created to match individual job types (e.g. so Plaso can have worker nodes that have more cpu than workers for other job types). Right now it's one global worker pool.
From a GKE autoscaling perspective, we can consider:
- 1 Large Task gets a single node for processing
- Multiple small tasks can run multiple pods on a single node
Also handy resource: https://medium.com/google-cloud/scale-your-kubernetes-cluster-to-almost-zero-with-gke-autoscaler-9c78051cbf40
Tagging 'celery' because I think we can use built-in functionality for this