flytekit icon indicating copy to clipboard operation
flytekit copied to clipboard

Feature/array node workflow parallelism

Open pvditt opened this issue 1 year ago • 1 comments

Tracking issue

https://github.com/flyteorg/flyte/issues/3924

Why are the changes needed?

Want to add support for array node map tasks to utilize threads under the parent workflow's set parallelism. Decrease confusion on default/expected map task parallelism

What changes were proposed in this pull request?

Set default parallelism to -1 to indicate

How was this patch tested?

Setup process

Screenshots

Check all the applicable boxes

  • [x] I updated the documentation accordingly.
  • [x] All new and existing tests passed.
  • [x] All commits are signed-off.

Related PRs

Needs https://github.com/flyteorg/flyte/pull/5062 to be merged first

Docs link

pvditt avatar Mar 15 '24 08:03 pvditt

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 53.57%. Comparing base (f4d894a) to head (451fd03). Report is 22 commits behind head on master.

:exclamation: Current head 451fd03 differs from pull request most recent head 3f596c2. Consider uploading reports for the commit 3f596c2 to get more accurate results

Additional details and impacted files
@@             Coverage Diff             @@
##           master    #2268       +/-   ##
===========================================
- Coverage   83.09%   53.57%   -29.52%     
===========================================
  Files         324      180      -144     
  Lines       24860    18133     -6727     
  Branches     3761     3547      -214     
===========================================
- Hits        20657     9715    -10942     
- Misses       3582     7991     +4409     
+ Partials      621      427      -194     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Mar 15 '24 08:03 codecov[bot]

Can you fix the unit test failure? The error:

FAILED tests/flytekit/unit/core/test_array_node_map_task.py::test_inputs_outputs_length - AssertionError: assert equals failed
  'tests.flytekit.unit.core.test_  'tests.flytekit.unit.core.test_ 
  array_node_map_task.map_many_in  array_node_map_task.map_many_in 
  puts_-6b^3bd0^353da5d^e6e8-4d79-82921  puts_b^f51001578d0^ae197a5+2c0af0a 
  -ead2b3-arraynode'                +99d^d89-arraynode'

eapolinario avatar Apr 18 '24 18:04 eapolinario

Can you fix the unit test failure? The error:

FAILED tests/flytekit/unit/core/test_array_node_map_task.py::test_inputs_outputs_length - AssertionError: assert equals failed
  'tests.flytekit.unit.core.test_  'tests.flytekit.unit.core.test_ 
  array_node_map_task.map_many_in  array_node_map_task.map_many_in 
  puts_-6�b^3b�d0^353d�a5d^e6e�8-4d7�9-82921�  puts_b^f51001578�d0^ae197�a5+2c0af0a� 
  -ead2b3�-arraynode'                +99�d^d�89-arraynode'

@eapolinario oof on my end. Just updated - will re-request your review after the builds pass

pvditt avatar Apr 18 '24 19:04 pvditt