Ben Sherman

- The array task handler sets the `task` field to the first task in the array. In other words, it assumes that all tasks in the array have the same...

~Currently, if the array executor submits a partial batch at the end, and some of those tasks fail and need to be retried, the array executor will hang because it...

If a task fails and is retried with increased resources, it will be batched with other tasks that may still be on their first attempt. In that case, the array...

We could also just provide config options for these things: - `executor.$array.groupKeys` (default: `['process', 'attempt']`) controls how batches are separated - `executor.$array.requestMaxResources` controls whether the array executor "plays if safe"...

The point of these config options is that there is a trade-off between bandwidth and latency when batching tasks like this, so users should ideally have the ability to manage...

- ~We may need to fetch the child job ids in order to check their status, in cases where the job id doesn't have it's own status.~ UPDATE: looks like...

With the latest commit I was able to run a test pipeline with SLURM array jobs 🎉 Now I will extend support to other executors, although I won't be able...

Thanks @abhi18av , that would be great. We're going to do some refactoring and then I'll ping you when the PR is closer to a final draft.

@pditommaso While array jobs are highly requested, there is further demand for "true" task grouping in which a group of tasks are executed on the same resources, especially in the...

The task grouping would probably work similarly to this PR -- instead of calling each `.command.run` in an array, you would create a single `.command.run` and call each `.command.sh` in...