[Core Feature] Use 'parallelism' in array tasks (and use it to improve performance)

[katrogan]

Motivation: Why do you think this is important? Parallelism can be requested by users to limit the number of concurrent array tasks. It would be nifty if we could use the statically defined parallelism value at run-time to batch up processing of inputs without requiring creating and tearing down a new pod for every invocation of a map task.

Goal: What should the final outcome look like, ideally? For example, with parallelism == 4 and inputs [1, 2, 3, ..., 15] array task execution could be broken up so that only 4 pods are broken up with the input batched semi-equally amongst them like so:

pod 0: [1, 5, 9, 13] pod 1: [2, 6, 10, 14] pod 2: [3, 7, 11, 15] pod 3: [4, 8, 12] (using a mod distribution)

or pod 0: [1, 2, 3, 4] pod 1: [5, 6, 7, 8] pod 2: [9, 10, 11, 12] pod 3: [13, 14, 15] (as an alternate allocation)

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

[Optional] Propose: Link/Inline OR Additional context This will require modifying the command line for array tasks in flytekit and pod construction in plugins to handle batching the inputs appropriately.

[katrogan]

cc @kumare3

[github-actions[bot]]

Hello 👋, This issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will close the issue if we detect no activity in the next 7 days. Thank you for your contribution and understanding! 🙏

[hamersaw]

Commenting to keep open.

[github-actions[bot]]

Hello 👋, this issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will engage on it to decide if it is still applicable. Thank you for your contribution and understanding! 🙏

[github-actions[bot]]

Hello 👋, This issue has been inactive for over 9 months and hasn't received any updates since it was marked as stale. We'll be closing this issue for now, but if you believe this issue is still relevant, please feel free to reopen it. Thank you for your contribution and understanding! 🙏