Allow users to specify batches per worker

[vishalbollu]

trafficstars

Description

Currently, clients must specify the number of workers required for the Job in the Job Submission. If the client doesn't know the total number of batches, they may over/under provision the required for number of workers for the job to complete in a timely manner. Allow clients to specify BatchesPerWorker to scale the number of workers based on the number batches in the job. Make max_workers a required field to prevent the users from provisioning an unexpected number of workers (if there is a mistake in the job submission).

A proposed solution for worker schema in the job submission might be:

"workers": {
  "desired":
  "batches_per_worker":
  "max_workers": 
}

Validations:

• only desired or batches_per_worker is specified but not both
• if batches_per_worker is specified, max_workers must be specified

Motivation

• Client can provide the desired number of workers to complete the job in a timely manner without figuring the number of batches in the job.