flink-connector-aws icon indicating copy to clipboard operation
flink-connector-aws copied to clipboard

[FLINK-35660] Adding parent-child shard ordering for DDB Streams source

Open gguptp opened this issue 7 months ago • 0 comments

Purpose of the change

  • Implements parent-child shard ordering in DDB Streams source
  • Source enumerator will only assign splits that correspond to a shard when parent shard is either finished, expired or doesn't exist
  • Source enumerator also discovers inconsistencies in the shard graph due to describestream and fixes it

Verifying this change

Added unit tests

Significant changes

(Please check any boxes [x] if the answer is "yes". You can first publish the PR and check them afterwards, for convenience.)

  • [ ] Dependencies have been added or upgraded
  • [ ] Public API has been changed (Public API is any class annotated with @Public(Evolving))
  • [x] Serializers have been changed
  • [x] New feature has been introduced
    • If yes, how is this documented? (not applicable / docs / JavaDocs / not documented)

gguptp avatar Jul 18 '24 12:07 gguptp