aws-cli

aws-cli copied to clipboard

Describe the bug

The pagination options emitted by --generate-cli-skeleton for list-functions are different from its actual invocation

aws lambda list-functions --generate-cli-skeleton
{
    "MasterRegion": "",
    "FunctionVersion": "ALL",
    "Marker": "",
    "MaxItems": 0
}

$ AWS_PAGER=cat aws lambda list-functions help | sed -ne "/^.*SYNOPSIS/,/^.*OPTIONS/p"
SYNOPSIS
            list-functions
          [--master-region <value>]
          [--function-version <value>]
          [--max-items <value>]
          [--cli-input-json | --cli-input-yaml]
          [--starting-token <value>]
          [--page-size <value>]
          [--generate-cli-skeleton <value>]

Observe the --starting-token NOT --marker

Expected Behavior

That the --generate-cli-skeleton would be the programmatic representation of the input arguments

Current Behavior

see above

Reproduction Steps

see above (or below)

Possible Solution

No response

Additional Information/Context

I can reproduce this with the docker image, so it is not just on my machine:

$ docker run --rm amazon/aws-cli:2.5.4 lambda list-functions --generate-cli-skeleton
{
    "MasterRegion": "",
    "FunctionVersion": "ALL",
    "Marker": "",
    "MaxItems": 0
}
$ docker run --rm amazon/aws-cli:2.5.4 lambda list-functions help | sed -ne '/^.*SYNOP/,/^.*OPTION/p'
SYNOPSIS
            list-functions
          [--master-region <value>]
          [--function-version <value>]
          [--max-items <value>]
          [--cli-input-json | --cli-input-yaml]
          [--starting-token <value>]
          [--page-size <value>]
          [--generate-cli-skeleton <value>]

CLI version used

aws-cli/2.5.4 Python/3.9.12 Linux/5.13.0-39-generic source/x86_64.ubuntu.21 prompt/off

Environment details (OS name and version, etc.)

Ubuntu 21.10, "brew install -s awscli"

Hi @mdaniel thanks for reaching out. The model for this paginator is defined here. There are some inconsistencies in how service APIs name their markers/tokens. Marker is a request parameter in the underlying API but it is abstracted to NextMarker for pagination.

The --starting-token is described in the documentation for this command here:

A token to specify where to start paginating. This is the NextToken from a previously truncated response.

So the --starting-token is only needed if there was a NextToken from the previous response. For general information on pagination using the CLI you can refer to this page in the User Guide.

I also created a tracking issue here for improvements to the functionality/documentation of --generate-cli-skeleton. We can include the pagination behavior as something that could be better documented as it relates to --generate-cli-skeleton. I hope that helps, please let us know if you have any questions or feedback.

Thank you for linking to the underlying issue

However, through what mechanism can a user learn what value to put into {"Marker":""} since the list-function output contains {"NextToken":""}? edit: well, I guess given the experiment below, the answer is to only use one style or the other, --style or --cli-input-json && --generate-cli-skeleton and not try and mix the two :-/

I thought perhaps you meant --generate-cli-skeleton was merely misconfigured but I had bad outcomes from some "just try it"s:

$ aws lambda list-functions --max-items 2
{
    "Functions": [
        {
            "FunctionName": ...
        }
    ],
    "NextToken": "eyJNYXJrZXIiOiBudWxsLCAiYm90b190cnVuY2F0ZV9hbW91bnQiOiAyfQ=="
}

$ aws lambda list-functions --cli-input-json '{"StartingToken": "eyJNYXJrZXIiOiBudWxsLCAiYm90b190cnVuY2F0ZV9hbW91bnQiOiAyfQ=="}'

Parameter validation failed:
Unknown parameter in input: "StartingToken", must be one of: MasterRegion, FunctionVersion, Marker, MaxItems

$ aws lambda list-functions --cli-input-json '{"Marker": "eyJNYXJrZXIiOiBudWxsLCAiYm90b190cnVuY2F0ZV9hbW91bnQiOiAyfQ=="}'

An error occurred (InvalidParameterValueException) when calling the ListFunctions operation: Invalid marker value: eyJNYXJrZXIiOiBudWxsLCAiYm90b190cnVuY2F0ZV9hbW91bnQiOiAyfQ==

While testing this, I found even more weird behavior; if one uses --max-items one gets {"NextToken":""} in the output, but if one uses {"MaxItems":2} the response contains the {"NextMarker":""} as documented by the OUTPUT section of the help

$ aws lambda list-functions --cli-input-json '{"MaxItems": 2}'
{
    "NextMarker": "bAsE64==",
    "Functions": [
        {}
    ]
}

I think this comment on an older issue might help clear up some of the behavior around pagination: https://github.com/aws/aws-cli/issues/2555#issuecomment-295825435

For more information on --generate-cli-skeleton and --cli-input-json I would refer to this page in the User Guide: https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-skeleton.html

I couldn’t reproduce the behavior you described from running this command: aws lambda list-functions --cli-input-json '{"MaxItems": 2}'. If you provide the debug logs by adding --debug to the command then we could look into it further.

based on --debug, the major difference is that --max-items filters client side whereas {"MaxItems":2} filters them server side

$ aws --debug --region ca-central-1 lambda list-functions --cli-input-json '{"MaxItems":2}' 2>github-6870.err.log | jq '.|keys'
[
  "Functions",
  "NextMarker"
]
$ aws --debug --region ca-central-1 lambda list-functions --max-items 2 2> github-6870.dash-args.log | jq '.|keys'
[
  "Functions",
  "NextToken"
]

github-6870.err.log github-6870.dash-args.log

Checking back in here. As mentioned previously there are inconsistencies between how service APIs define their pagination tokens. The Lambda ListFunctions API uses NextMarker as you demonstrated but the CLI pagination uses NextToken. And your example does show the server-side vs client-side distinction in behavior.

You could disable the CLI pager as described in the documentation if that would help with your use case. If you have any feedback on how the CLI pagination documentation could be improved please let us know here. And for improvements regarding generate-cli-skeleton we can continue tracking those in https://github.com/aws/aws-cli/issues/6695 if you want to add any comments to that issue.

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.