flytekit
flytekit copied to clipboard
[WIP] Add support to ArrayNode to map over Launch Plans
Tracking issue
Why are the changes needed?
ArrayNode currently only supports mapping over a subset of flyte entities (PythonFunctionTasks). To map over other flyte entities such as LaunchPlans we need a new Flytekit node type as the ArrayNodeMapTask implementation can only support Python tasks.
What changes were proposed in this pull request?
Create new ArrayNode class to map over launch plans. Will have a follow up PR to have this new ArrayNode class support Python tasks (as per discussion with @eapolinario and @wild-endeavor).
How was this patch tested?
Ran in sandbox with private fork branch: https://github.com/unionai/flyte/pull/255
Setup process
@task
def multiply(val: int, val1: int) -> int:
# if val == 1:
# raise Exception("This is a test exception")
return val * val1
@workflow
def parent_wf(num: int, num1: int) -> int:
return multiply(val=num, val1=num1)
ex_lp1 = LaunchPlan.get_default_launch_plan(current_context(), parent_wf)
@workflow
def mapped_lp1() -> list[int]:
a = map_task(ex_lp1, min_success_ratio=0.6)(num=[1, 3, 5], num1=[2, 4, 6])
return a
Screenshots
Check all the applicable boxes
- [ ] I updated the documentation accordingly.
- [ ] All new and existing tests passed.
- [x] All commits are signed-off.
Related PRs
https://github.com/unionai/flyte/pull/255