aws-sdk-pandas
aws-sdk-pandas copied to clipboard
Support `next_token` in paginated requests to `timestream.query()`
Is your idea related to a problem? Please describe.
Support next_token
in paginated requests to timestream.query()
Describe the solution you'd like
Allow calls to timestream.query()
to support paginated return of DataFrames with a next_token
to allow client to manage pagination of data returned from timestream.
P.S. Please do not attach files as it's considered a security risk. Add code snippets directly in the message body as much as possible.
It should be possible to add the continuation token into the attrs
property of Pandas. Unfortunately, the docs says that the feature is experimental and could change at any time.
Another issue is that the boto3 paginator doesn’t return the pagination token either. So essentially we’d need to write our own paginator just to get the continuation token.
@LeonLuttenberger @cnfait Each page here has QueryId
and NextToken
. We can change _rows_to_df
to add attributes to the data frame it produces if any are passed and pass those attributes if chunked=True