Temporal spacing of task submitting

[jhclark]

Currently, jobs are submitted (e.g. to qsub) as soon as they become available. Especially when a workflow begins to run, this can result in a very large number of calls to qsub at once. Often, schedulers don't behave well under this sort of load and produce spurious error messages.

To work around this limitation of schedulers, we will implement a mechanism for Submitters to keep some sort of global state (note: under no circumstances should this be done with static variables).

We might consider doing this using one instance of each submitter object per submitter definition in the workflow. This way we can manually add a timeout between submits. It would be best if we could find a way of delegating this back to the bash submitter code somehow to avoid having too many decisions hardcoded in Scala.