DynamoDB Table Resource should support Pagination for Scan and Query

[rbu]

The boto3 "Table" Resource API does not provide easy access to paged results when using Scan or Query actions. That's unlike other high level boto3 Resource APIs like S3 which supports s3.Bucket('bucket').objects.pages().

The underlying DynamoDB client supports pagination, which leaves the boto3 user with the choice between nice attribute access + ugly pagination or ugly attribute access + nice pagination, which is unfortunate.

[swetashre]

@rbu - Thank you for your post. I will mark this as feature request.

[tomislacker]

@rbu Thank you much for this, I'm also interested. I'm surprised by there being relatively little activity on this issue. But I suppose if the consumers of DynamoDB that aren't using some sort of ORM already are the main ones affected; I'm not really surprised. They're already dealing a lot with some gnarly JSON blobs.

[kevindixon]

Any update on this feature request - is it planned at all?

[systematicguy]

What is holding up this issue? Could I help?

[aBurmeseDev]

Thank you all for checking in and apologies for the delay.

We created a backlog item to look into this feature request and if anyone wants to share additional use cases or feedback then please feel free to leave a comment. Although we can't give a definite timeline at the moment, we'll make sure to keep it posted as soon as we have an update and please feel free to check back here in the future.

[mrtj]

I use a very simple implementation:

from typing import Callable, Dict, Iterator

def dynamodb_paginator(action: Callable, kwargs: Dict) -> Iterator[Dict]:
    while True:
        response = action(**kwargs)
        for item in response["Items"]:
            yield item
        if "LastEvaluatedKey" in response:
            kwargs["ExclusiveStartKey"] = response["LastEvaluatedKey"]
        else:
            break

Usage:

session = boto3.Session()
dynamodb = session.resource("dynamodb")
table = dynamodb.Table("my_table")
scan_kwargs = { "ScanFilter": ... }  # set here the arguments you would use in the scan() method call
for item in dynamodb_paginator(table.scan, scan_kwargs):
    ... # do something with the item