list_shards paginator broken for 1000+ shards

[slydon]

When using the paginator in list_shards for a stream with over 1000 shards an exception is thrown because the list_shards API expects the StreamName to not be set when specifying the NextToken parameter.

This is related to https://github.com/boto/botocore/issues/1462#issuecomment-567845338

[swetashre]

@slydon - Thank you for your post. I am not able to reproduce the issue. The comment you linked does not say anything about the exact error. Can you please provide me code sample you are using with the exact error you are getting ?

This is the documentation on how to use paginator with list_shards api: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/kinesis.html#Kinesis.Paginator.ListShards

[slydon]

The error is:

botocore.errorfactory.InvalidArgumentException: An error occurred (InvalidArgumentException) when calling the ListShards operation: NextToken and StreamName cannot be provided together.

The code sample is:

import boto3
k = boto3.client('kinesis')
ls_p = k.get_paginator('list_shards')
len([s for p in ls_p.paginate(StreamName='large') for s in p.get('Shards', [])])

How did you try to reproduce this? If you are using kinesalite it doesn't implement this functionality correctly. https://github.com/mhart/kinesalite/blob/master/actions/listShards.js#L12

[swetashre]

@slydon - Sorry i have directly called the paginator with StartingToken that's why did not able to reproduce the issue. But now i am able to reproduce the issue with the below code:

paginator = kinesis.get_paginator("list_shards").paginate(StreamName="test-stream", PaginationConfig={"PageSize": 1})

for page in paginator:
      print(page)

Marking this as bug.

[bbayles]

I ran into this issue also.

Here is a code that can be used until the problem is fixed:

def list_shards(kinesis_client, stream_name):
    all_shards = []
    kwargs = {'StreamName': stream_name}
    while True:
        resp = kinesis_client.list_shards(**kwargs)
        for shard in resp['Shards']:
            all_shards.append(shard)

        next_token = resp.get('NextToken')
        if next_token:
            kwargs = {'NextToken': next_token}
        else:
            break
    
    return {'Shards': all_shards}

[adev-code]

Hello, the Service Team has mentioned that this has been fixed. I have tried (for testing):

import boto3

def list_shards(stream_name, page_size=1):
    client = boto3.client('kinesis')
    paginator = client.get_paginator('list_shards')

    shards = []
    for page in paginator.paginate(StreamName=stream_name, PaginationConfig={'PageSize': page_size}):
        shards.extend(page.get('Shards', []))

    return shards

shards = list_shards('foo', page_size=1)
print(shards)

and it returns all shards in one page.

[github-actions[bot]]

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.