amazon-kinesis-client icon indicating copy to clipboard operation
amazon-kinesis-client copied to clipboard

ProcessRecordsInput.isAtShardEnd value is not passed to processRecords

Open ktulinger opened this issue 2 years ago • 1 comments

Hello,

we identified a possible bug in handling the isAtShardEnd value for a record processor. When receiving the records in processRecords, the value of isAtShardEnd is always false.

After looking at the code base, it is never set in the builder object: https://github.com/awslabs/amazon-kinesis-client/blob/bedae95db96f661cc1d3bb19cfa96fc5cb131201/amazon-kinesis-client/src/main/java/software/amazon/kinesis/lifecycle/ProcessTask.java#L217-L218

We use alpakka kinesis library, which, we observed, relies on the shardEnd value to be correctly set in order to properly checkpoint the shards.

ktulinger avatar Mar 30 '22 11:03 ktulinger

Akka's Alpakka Kinesis library relies on this functionality, see their ShardProcessor: https://github.com/akka/alpakka/blob/master/kinesis/src/main/scala/akka/stream/alpakka/kinesis/impl/ShardProcessor.scala

The library is packaged as akka-stream-alpakka-kinesis.

sochi avatar Mar 30 '22 11:03 sochi

Thanks for your contribution! PR merged; closing issue.

This should be included in our next release which will likely be version 2.4.6.

stair-aws avatar Jan 31 '23 18:01 stair-aws