powertools-lambda-java icon indicating copy to clipboard operation
powertools-lambda-java copied to clipboard

Feature Request: Do No Continue Processing Event in Batch Mode for Kinesis/DDBStreams

Open belugabehr opened this issue 8 months ago • 11 comments

When processing a DDB Stream in batch mode, I want to stop processing when a failure is reached. Since this is a stream, and ordering of message is important for me, the processing should immediately stop.

That is to say, if my data is partitioned on Purchase ID, I want to ensure all events related to the same purchase are played in order. If a failure occurs, the processing of the stream should stop and retry later.

Expected Behavior

When an error occurs, the offending event should be checkpointed, and processing should stop.

https://docs.aws.amazon.com/lambda/latest/dg/services-ddb-batchfailurereporting.html

Current Behavior

For DDB Stream Batch processing, the stream will continue to be reprocessed, and the same messages will be repeated again, and again.

https://github.com/aws-powertools/powertools-lambda-java/blob/8dcdddf025619cd8c5e50624e981454b22cd6fa3/powertools-batch/src/main/java/software/amazon/lambda/powertools/batch/handler/DynamoDbBatchMessageHandler.java#L59-L75

Possible Solution

Return on any error. Take a look at the following example as reference:

https://docs.aws.amazon.com/lambda/latest/dg/services-ddb-batchfailurereporting.html

            } catch (Exception e) {
                /* Since we are working with streams, we can return the failed item immediately.
                   Lambda will immediately begin to retry processing from this failed item onwards. */
                batchItemFailures.add(new StreamsEventResponse.BatchItemFailure(curRecordSequenceNumber));
                return new StreamsEventResponse(batchItemFailures);
            }

or a little bit nicer:

                return new StreamsEventResponse(Collections.singletonList(
                        new StreamsEventResponse.BatchItemFailure(curRecordSequenceNumber)));

belugabehr avatar Apr 16 '25 16:04 belugabehr