PageSize parameter for dynamodb.Client.Query

[nate-anderson]

Describe the feature

The AWS CLI tool for DynamoDB allows specifying a page size (--page-size) which is the maximum number of matching items returned by the query. This is different from the limit option, which limits the number of items evaluated, including those which are not returned to the client.

dynamodb.QueryInput should export a PageSize property which, when set, limits the number of items returned to the caller, independent of the number of items evaluated.

Use Case

Pagination with DynamoDB in Go is currently very difficult. Here's an example function

func ListBooks(ctx context.Context, limit int, paginationToken string, paginationKey map[string]types.AttributeValue, client *dynamodb.Client) ([]User, map[string]types.AttributeValue, error) {
	filterExpr := "#active = true"
	result, err := client.Query(ctx, &dynamodb.QueryInput{
		TableName:         &tableName,
		ExclusiveStartKey: paginationKey,
		FilterExpression:  &filterExpr,
		ExpressionAttributeNames: map[string]string{
			"#active": "active",
		},
	})

	if err != nil {
		return []User{}, nil, err
	}

	users := make([]User, len(result.Items))
	err = attributevalue.UnmarshalListOfMaps(result.Items, &users)
	return users, result.LastEvaluatedKey, err
}

I see no way to use the limit argument without either a) potentially returning more rows than the caller asked for, or b) limiting the returned rows by slicing them off myself and potentially missing them when the caller returns for the next page, because the LastEvaluatedKey returned by this query will be past the results I slice off.

Proposed Solution

A PageSize field should be added to the dynamodb.QueryInput struct, corresponding to the --page-size option used by the AWS CLI.

Other Information

No response

Acknowledgements

• [ ] I may be able to implement this feature request
• [ ] This feature might incur a breaking change

AWS Go SDK V2 Module Versions Used

github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/[email protected] github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/feature/ec2/[email protected] github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/[email protected] github.com/aws/aws-sdk-go-v2/[email protected] github.com/google/[email protected] github.com/aws/aws-sdk-go-v2/feature/ec2/[email protected] github.com/aws/[email protected] github.com/aws/aws-sdk-go-v2/feature/ec2/[email protected] github.com/aws/[email protected] github.com/aws/aws-sdk-go-v2/feature/ec2/[email protected] github.com/google/[email protected] github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/[email protected] github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/internal/[email protected] github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/internal/endpoints/[email protected] github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/[email protected] github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/[email protected] github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/internal/[email protected] github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/internal/endpoints/[email protected] github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/service/internal/[email protected]

Go version used

go version go1.18.1 linux/amd64

[danielkhtse]

Would it be added soon?

[aajtodd]

Both the CLI and AWS SDK for Go work off the same underlying DynamoDB Query API

--page-size is mapped to Limit in the CLI:

> aws dynamodb query --table-name movies --key-condition-expression "#y = :v1" --expression-attribute-values '{":v1":{"N": "2013"}}' --expression-attribute-names  "#y = year"  --page-size 210 --debug

...
2023-02-21 11:48:41,078 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=Query) with params: {'url_path': '/', 'query_string': '', 'method': 'P
OST', 'headers': {'X-Amz-Target': 'DynamoDB_20120810.Query', 'Content-Type': 'application/x-amz-json-1.0', 'User-Agent': 'aws-cli/2.9.21 Python/3.11.1 Darwin/22.3.0 source/x86_
64 prompt/off command/dynamodb.query'}, 'body': b'{"TableName": "movies", "KeyConditionExpression": "#y = :v1", "ExpressionAttributeNames": {"#y": "year"}, "ExpressionAttribute
Values": {":v1": {"N": "2013"}}, "Limit": 210}', 'url': 'https://dynamodb.us-west-2.amazonaws.com/', 'context': {'client_region': 'us-west-2', 'client_config': <botocore.config
.Config object at 0x110c53190>, 'has_streaming_input': False, 'auth_type': None}}   

...

NOTE: The Limit: 210 in the request body that was mapped from the --page-size CLI argument.

I'd recommend using the provided Query paginator. DDB will never process more items than the Limit so the number of matching results will be <= the limit.

Closing.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.