Ashishonce/multiplerunmultiprocessing

Open ashishonce opened this issue 5 years ago • 3 comments

This PR is for supporting multiple run submissions as part of a single WF execution. we are using multiprocessing library to process each entry here. to save the time on parallel waits for execution to complete. currently creating a pool of processes as 8, but once we agree on this approach we can find an optimal value of processes to start.

Apr 23 '20 13:04 ashishonce

added Unit Test basic one, if this format looks fine we will go ahead with that

Apr 27 '20 09:04 ashishonce

@ashishonce this looks good, but we are going to wait to merge until we finish the integration tests across all the actions. This should be finished this week.

Apr 27 '20 16:04 awmatheson

@awmatheson that makes sense, but just a point. This PR is a breaking change as it changes the run.json format which is a list now. So your integrations tests will still need to update then.

Apr 27 '20 16:04 ashishonce