Repeated empty segments commits for Kinesis merged Shards

[noob-se7en]

Problem When shards are merged or reduced in Kinesis, Servers send segment consumed message to Controller with reason: endOfPartition reached. Controller during commit process can incorrectly determine partition has not reached its end and creates new consuming segment. The servers immediately commit that new segment. This process keeps on repeating.

Error log:

2025/11/11 18:41:00.041 WARN [KinesisStreamMetadataProvider] [grizzly-http-server-0]  Reached max attempts to check if end of shard reached from checkpoint {"shardId-000000000000":"<>"}.  Assuming we have not consumed till end of shard.

Code to be re-visited: https://github.com/apache/pinot/blob/844d4ab8f9fcdc8fa71ffd750ac31ee858a815d0/pinot-plugins/pinot-stream-ingestion/pinot-kinesis/src/main/java/org/apache/pinot/plugin/stream/kinesis/KinesisStreamMetadataProvider.java#L228-L257