maestrowf icon indicating copy to clipboard operation
maestrowf copied to clipboard

Add the ability to run a given step in a workflow

Open jsemler opened this issue 7 years ago • 4 comments

It would be useful to have the ability to run a given step or a range of steps in the workflow. This would be really helpful for debugging a workflow.

jsemler avatar Jul 14 '17 04:07 jsemler

So this leads to a new set of questions/circumstances --

  1. If you specify a step in the workflow, you would have to run the steps preceding it.
  2. Along with that, would you want the workflow to rerun all the preceding steps if they've been run previously? There's a certain amount of scoping I'm aiming for in how much Maestro needs to know about codes running it. I think it's overkill to try and determine if a step needs to be rerun based on artifacts... Though if you want the base assumption to be that all steps prior are okay to reuse that may be okay. That'd take some thought on how to elegantly manage it.

What expected behavior do you have around selecting the step, I guess is the real question.

FrankD412 avatar Jul 14 '17 04:07 FrankD412

For this case, I just want the step I specify to run. The need to run preceding steps could potentially be solved by being able to specify a range (run step 1 to step 5). Then Maestro would only need to manage running from the given steps. If a given step had already been run, I think the expected behavior would be to rerun it. Maestro might still have other internal tracking it would need to do, though I'm not sure how complex that would be.

jsemler avatar Jul 14 '17 21:07 jsemler

I agree it would be nice to be able to run only one name in the yaml file, especially when prototyping

doutriaux1 avatar Jan 06 '20 18:01 doutriaux1

@doutriaux1 -- this is where it gets tricky. Because Maestro currently can't restart a study, running to a given name would mean running all dependent tasks because a study under the hood is a DAG. So if you end up repeatedly testing, that means you'll end up testing dependent steps over and over.

FrankD412 avatar Jan 06 '20 18:01 FrankD412