Feature request: Do No Continue Processing Event in Batch Mode for Kinesis/DDBStreams

[hjgraca]

Use case

When processing a DDB Stream in batch mode, I want to stop processing when a failure is reached. Since this is a stream, and ordering of message is important for me, the processing should immediately stop.

That is to say, if my data is partitioned on Purchase ID, I want to ensure all events related to the same purchase are played in order. If a failure occurs, the processing of the stream should stop and retry later.

Initial issue and more information: https://github.com/aws-powertools/powertools-lambda-java/issues/1820

Solution/User Experience

Need to review the Error Handling policy to see if we already support this feature https://docs.powertools.aws.dev/lambda/dotnet/utilities/batch-processing/#error-handling-policy

Alternative solutions

Acknowledgment

• [x] This feature request meets Powertools for AWS Lambda (.NET) Tenets
• [ ] Should this be considered in other Powertools for AWS Lambda languages? i.e. Python, Java, and TypeScript

[lachriz-aws]

I would argue that we already support this feature request using the ErrorHandlingPolicy setting.

If any type of Exception occurs during record processing, we check the ErrorHandlingPolicy to decide what to do. If it equals StopOnFirstBatchItemFailure then we throw a CircuitBreakerException, see:

https://github.com/aws-powertools/powertools-lambda-dotnet/blob/dfb8ae4272fd7047d3ed8909956c4cde712dff0a/libraries/src/AWS.Lambda.Powertools.BatchProcessing/BatchProcessor.cs#L218-L225

... which causes the outer loop (the one that iterates the batch) to stop iterating further, see:

https://github.com/aws-powertools/powertools-lambda-dotnet/blob/dfb8ae4272fd7047d3ed8909956c4cde712dff0a/libraries/src/AWS.Lambda.Powertools.BatchProcessing/BatchProcessor.cs#L125-L128

Important note from the docs about when this feature is used together with parallel processing:

When using StopOnFirstBatchItemFailure and parallel processing is enabled, all batch items already scheduled to be processed, will be allowed to complete before the batch processing stops. Therefore, if order is important, it is recommended to use sequential (non-parallel) processing together with this value.

[hjgraca]

Thank you @lachriz-aws , really appreciate the feedback. I agree and that is my understanding as well.

I need to review with other runtimes to confirm if it covers all use cases.