maestrowf
maestrowf copied to clipboard
Add the ability to run a given step in a workflow
It would be useful to have the ability to run a given step or a range of steps in the workflow. This would be really helpful for debugging a workflow.
So this leads to a new set of questions/circumstances --
- If you specify a step in the workflow, you would have to run the steps preceding it.
- Along with that, would you want the workflow to rerun all the preceding steps if they've been run previously? There's a certain amount of scoping I'm aiming for in how much Maestro needs to know about codes running it. I think it's overkill to try and determine if a step needs to be rerun based on artifacts... Though if you want the base assumption to be that all steps prior are okay to reuse that may be okay. That'd take some thought on how to elegantly manage it.
What expected behavior do you have around selecting the step, I guess is the real question.
For this case, I just want the step I specify to run. The need to run preceding steps could potentially be solved by being able to specify a range (run step 1 to step 5). Then Maestro would only need to manage running from the given steps. If a given step had already been run, I think the expected behavior would be to rerun it. Maestro might still have other internal tracking it would need to do, though I'm not sure how complex that would be.
I agree it would be nice to be able to run only one name
in the yaml file, especially when prototyping
@doutriaux1 -- this is where it gets tricky. Because Maestro currently can't restart a study, running to a given name would mean running all dependent tasks because a study under the hood is a DAG. So if you end up repeatedly testing, that means you'll end up testing dependent steps over and over.