[Feature]: Support for keepLastN in DO_WHILE Task

[sriharimalagi]

Discussed in https://github.com/orgs/conductor-oss/discussions/134

^{Originally posted by sriharimalagi April 16, 2024} Problem statement:

• We are running DO_WHILE tasks for large number of iterations (>1000). When each DO_WHILE task iteration is completed, the output is stored into the Task.outputData.
• Since the results after each iteration is appended, the Task Output Max Threshold limit is breached, causing the workflow to get terminated.
• We would like to apply a solution such that previously executed iterations' output can be discarded.

Analysis:

• Upon reviewing the DO_WHILE task documentation, I have identified a property named keepLastN which is currently not supported in the open source code.
• Leveraging this logic for our DO_WHILE task would be beneficial. I am prepared to initiate a pull request for its implementation.

[jeffbulltech]

Assigned to @bradyyie as an enhancement for an upcoming release.