kinesis-sql icon indicating copy to clipboard operation
kinesis-sql copied to clipboard

Fix listShards to only specify nextToken

Open Anthuang opened this issue 2 years ago • 2 comments

This fixes two bugs:

  1. If listShards returns multiple pages, then we run into an infinite loop with while (!nextToken.isEmpty), because nextToken will never be set back to null because:
if (returnedToken != null) {
  nextToken = returnedToken
  1. listShards cannot allow both StreamName and NextToken to be specified together.

Anthuang avatar Oct 21 '21 19:10 Anthuang

@itsvikramagr this is tested against streams with >1k shards which forces paging on the Kinesis API

chadlagore avatar Oct 21 '21 19:10 chadlagore

Does this read the equal number of messages too ? because I see there is a data skew.

itsmesrds avatar Mar 20 '22 13:03 itsmesrds