`aws cognito-idp list-users` and `aws cognito-idp list-user-pools` do not paginate by default

[hauntingEcho]

When the --no-paginate option is not specified, pagination should be followed automatically to return the whole result set. However, aws cognito-idp list-users does not do this, nor does aws cognito-idp list-user-pools (which even has a required --max-results argument)

[hauntingEcho]

to clarify, this is using the version as installed via homebrew (aws-cli/1.16.80 Python/3.7.2 Darwin/17.7.0 botocore/1.12.70)

[seb-steuer]

+1

[klaytaybai]

Hi @hauntingEcho and @seb-steuer, it sounds like you have a couple asks here, so I’ll address them separately.

Ask 1: If the user doesn’t specify –no-paginate for aws cognito-idp list-users, their results should be paginated by default.

--no-paginate is actually not a valid option for this command at this time. Including or excluding it will do nothing. Pagination for this method in the CLI is currently opt-in up to 60 users, which mirrors the service’s API. You can do so by including the limit option: aws cognito-idp list-users –user-pool-id <value> --limit <value 0-60>. I verified that if your group has more than 60 users, pagination occurs by default, and you will get a pagination token in your response.

Ask 2: If the user doesn’t specify —no-paginate for aws cognito-idp list-user-groups, their results should be paginated by default.

This seems reasonable, especially considering that the CLI reference indicates that this should be the behavior. Unfortunately we seem to have a bug in our documentation for this command in the CLI reference. As you have noticed, the –max-results argument is required. This is because this is what the service API expects. We plan to get these docs cleaned up. We are considering trying to get the service API's --max-results to be made optional and defaulted to the current max value of 60 and/or to allow the –no-paginate option in the CLI. To help us make our decisions, we welcome any discussion of these possible features from you and others in the community.

[hauntingEcho]

Thanks for the response! The documentation I had been working from that led me to think that both calls should give all results by default was the second paragraph of The 'Pagination' subsection of 'Using the AWS CLI':

" By default, the AWS CLI uses a page size of 1000 and retrieves all available items. For example, if you run aws s3api list-objects on an Amazon S3 bucket that contains 3,500 objects, the CLI makes four calls to Amazon S3, handling the service-specific pagination logic for you in the background and returning all 3,500 objects in the final output. "

[klaytaybai]

That is how the paginators in the CLI work when implemented. We are hoping to make some changes to make this more consistent across the CLI, including these commands in cognito-idp. I can't make any guarantees on a timeline for this.

[no-response[bot]]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

[hauntingEcho]

@justnance - what response was needed from me on this issue?

By my reading the responses from @klaytaybai were clarification that this is actually two issues (for list-users and list-user-pools), acknowledgement that the issues are in fact issues, and a statement that this isn't necessarily in the timeline just yet.

[justnance]

@hauntingEcho - Sorry for the confusion and adding the 'closing-soon' label. Looks like this was labeled as a feature request.

[peterlgh7]

No updates on this? The documentation is misleading and the resulting behavior is not consistent with the rest of the aws-cli.

[kshitijrao]

Any Updates on Pagination Feature?

[johannesfloriangeiger]

Further to the above, when I want to filter user pools by name to automate my configuration I can only do so on the client side, by setting max results to 60 and "hoping" nobody created more than 60 user pools and my user pool can by found on page 2.

[hanna-becker]

Any updates here? Having to paginate manually every time we call the list-users endpoint is quite annoying. The problem is that we're only looking for one specific user most of the time, but sometimes receive an empty first page with a pagination token.

[tim-finnigan]

Checking in to get clarification as a few different topics have been discussed here.

Here is the AWS CLI pagination documentation for reference: https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-pagination.html

The underlying Cognito IDP API ListUsers returns a maximum limit of 60 results per request, which isn't something that the CLI can control directly. A NextToken can be used to return the next set of items in the list when there are more items to return. Is the request here for the Cognito team to increase the default limit to more than 60 per API call?

[hanna-becker]

Hello @tim-finnigan, thanks for your reply!

Let me explain what we'd like to achieve, and what is happening instead.

We are actually not using the cli, but the JS client, which I assume has the same behavior.

We have a user search on our website, which is a simple prefix search based on the first name. We want to display a maximum of 5 results. So we're using this code in a NodeJS based Lambda function:

  const listUsersParams = {
    UserPoolId: <userPoolId>,
    AttributesToGet: ['given_name', 'family_name'],
    Filter: `given_name ^= "${searchString}"`,
    Limit: 5,
  };

  const data = await cognitoIdentityServiceProvider.listUsers(listUsersParams).promise();

What I see now on one of our stages is that if I start the search with the letter "K" (we have a couple of test users whose first name starts with that letter on that stage) is that I get an empty result set. I see in the Lambda's logs that the query result looks something like this:

{
    "Users": [],
    "PaginationToken": <long_string>
}

So there are users, they're just not on the first page.

So what's the issue? I suppose the limit got applied before the filter? This is somehow unexpected. Obviously we would like the limit to be applied to the filtered result set.

I was hoping that using the --no-paginate option would work around this by accumulating results up to the specified limit before returning them to the client.

[tim-finnigan]

Hi @hanna-becker thanks for following up here. I think what you're describing is the expected behavior. The --no-paginate documentation notes:

The --no-paginate option disables following pagination tokens on the client side. When using a command, by default the AWS CLI automatically makes multiple calls to return all possible results to create pagination. One call for each page. Disabling pagination has the AWS CLI only call once for the first page of command results.

And in the cognito-idp list-users documentation it notes:

You can also list users with a client-side filter. The server-side filter matches no more than one attribute. For an advanced search, use a client-side filter with the --query parameter of the list-users action in the CLI. When you use a client-side filter, ListUsers returns a paginated list of zero or more users. You can receive multiple pages in a row with zero results. Repeat the query with each pagination token that is returned until you receive a null pagination token value, and then review the combined result.

For more information about server-side and client-side filtering, see FilteringCLI output in the Command Line Interface User Guide.

Have you tried using the --query parameter for client-side filtering?

[github-actions[bot]]

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

[hauntingEcho]

@tim-finnigan it looks like the response requested is not related to the ticket's request, but rather Hanna's separate --query issue, right?

[tim-finnigan]

Hi @hauntingEcho yes but I also added that label earlier for this question:

The underlying Cognito IDP API ListUsers returns a maximum limit of 60 results per request, which isn't something that the CLI can control directly. A NextToken can be used to return the next set of items in the list when there are more items to return. Is the request here for the Cognito team to increase the default limit to more than 60 per API call?

I'm still not sure if I fully understand the original request but this sounds like something that should redirected to the Cognito IDP team. We generally recommend reaching out through AWS Support for these service API related requests but we can also redirect them internally.

[hauntingEcho]

This ticket is only related to the response that the CLI returns not following the CLI's documentation as mentioned here. It sounds like the CLI may have to make multiple API requests to accomplish this given the current API, but wouldn't be blocked on API support and honestly the API probably shouldn't support unlimited-size calls itself.

[paul-uz]

@hanna-becker we see the same issue. Certainly seems like limit is taking priority over the filter. I suspect the underlying database operates like DynamoDB in how it handles these things. Frankly its a bit of a joke.