Beaker Executor should execute multiple Tango steps in one Beaker experiment

[dirkgr]

🚀 The feature, motivation and pitch

Beaker experiments have significant overhead. When we're running many small Tango steps, we can save some time by running multiple steps at once.

An issue is that we don't know ahead of time how long a step will take, so we don't know which steps make sense to group together. When running like this, it would make sense for Beaker experiments to poll the workspace for new work. This would be a drastic change in how we execute things, and it would overlap with Beaker functionality in uncomfortable ways. This feature requires some careful thought.

Alternatives

No response

Additional context

No response

[dirkgr]

The caveats in the description of this issue stand, but we can probably find sub-graphs of the whole execution graph that always make sense to group together into one Beaker experiment. A run of steps where each one depends on the previous step, and no other dependencies are unfulfilled, those could always be grouped. It would cut the overhead of my current experiment by 66%.

[epwalsh]

A run of steps where each one depends on the previous step, and no other dependencies are unfulfilled, those could always be grouped. It would cut the overhead of my current experiment by 66%.

That was my thought as well. These steps would also have to have matching step resource requirements.

[dirkgr]

These steps would also have to have matching step resource requirements.

Or we take the max of the resource requirements.