go-dynamodb-stream-subscriber icon indicating copy to clipboard operation
go-dynamodb-stream-subscriber copied to clipboard

Published 20 hours ago • verified publisher icon urakozz

checkpointing for streams

Open abhineshada opened this issue 5 years ago • 2 comments

  • Common approach to handle checkpointing while using GO sdks?
  • How to handle missing/duplicated data?

abhineshada avatar Apr 29 '19 14:04 abhineshada

What the hell is checkpointing? I didn't use DynamoDB for some years, is is new term there?

How to handle missing/duplicated data?

However you like, this library only wraps complexity of iterating over overengineered dynamo streams

urakozz avatar Apr 29 '19 14:04 urakozz

By checkpoint, I mean tracking the status of shards/data processed.

Refer: https://docs.aws.amazon.com/streams/latest/dev/kinesis-record-processor-ddb.html https://www.infoq.com/articles/demystifying-dynamodb-streams

KCL adapter for dynamoDB streams has checkpoint feature (only available in Java): https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.KCLAdapter.html

The library is really well written. I had few doubts regarding how data accuracy can be maintained using go SDK.

abhineshada avatar Apr 29 '19 17:04 abhineshada