metaflow icon indicating copy to clipboard operation
metaflow copied to clipboard

fix: Deterministic foreach task id's for Argo Workflows

Open saikonen opened this issue 1 year ago • 4 comments

changes the Task ID generation for tasks in foreach branches to be deterministic. This is so that the following join steps do not require an enormous list of ID's to be passed in through the input-paths input Parameter, which has limited the number of foreach tasks we can run on Argo Workflows until now.

Note: Still requires rigorous testing for different split cases to confirm this is ready for prime time.

Closes #1538 properly. Supersedes #1655

saikonen avatar Jan 27 '24 18:01 saikonen

Testing[608] @ d7a320bc6c1d4549f950e574a5afbddaef02bc8a

nflx-mf-bot avatar Jan 27 '24 18:01 nflx-mf-bot

Testing[608] @ d7a320bc6c1d4549f950e574a5afbddaef02bc8a had 1 FAILURE.

nflx-mf-bot avatar Jan 27 '24 20:01 nflx-mf-bot

I'll check but failure is probably that card test that I need to increase timeout for. There is no change to common code.

romain-intel avatar Jan 28 '24 07:01 romain-intel

Testing[608] @ bcc202639c392ee9aa5386cb1ef6b2c201a84c7f

nflx-mf-bot avatar Jan 31 '24 14:01 nflx-mf-bot