Cook icon indicating copy to clipboard operation
Cook copied to clipboard

Submit jobs directly to Kubernetes Scheduler by pool config

Open ahaysx opened this issue 2 years ago • 1 comments

This change adds a proof of concept for configuring pools to bypass Fenzo and submit their considerable jobs directly to Kubernetes. These jobs are scheduled by a new Kubernetes handler, which respects quota and rate limits. Pools not configured for this new handler are sent to Fenzo in the same way Cook does presently. Because of the new feature flag, this change should be deployable with no change in behavior.

  • New: make-pool-handler calls new handle-fenzo-pool, which is identical to old make-offer-handler except with some common variables refactored into the handler factory.
  • New: handle-kubernetes-pool and associated helper functions added and tested manually on-prem.
  • New: config to optionally have a pool be managed by the new Kubernetes handler.
  • New: some testing for helper functions supporting the Kubernetes handler.

Future work

  • Address TODOs.
  • Keep instance in Waiting state until it is initializing on a node
  • Update instance hostname after being scheduled by Kubernetes
  • Add any missing metrics needed for experiment measurements
  • Implement backpressure mechanism

ahaysx avatar Jul 27 '22 04:07 ahaysx

Will update tomorrow moving config to be under scheduler (one config for whole cook cluster) and to pass it in to create-datomic-scheduler (for testing).

ahaysx avatar Jul 27 '22 21:07 ahaysx