aws-sdk-pandas
aws-sdk-pandas copied to clipboard
Support `next_token` in paginated requests to `timestream.query()`
Is your idea related to a problem? Please describe.
Support next_token in paginated requests to timestream.query()
Describe the solution you'd like
Allow calls to timestream.query() to support paginated return of DataFrames with a next_token to allow client to manage pagination of data returned from timestream.
P.S. Please do not attach files as it's considered a security risk. Add code snippets directly in the message body as much as possible.
It should be possible to add the continuation token into the attrs property of Pandas. Unfortunately, the docs says that the feature is experimental and could change at any time.
Another issue is that the boto3 paginator doesn’t return the pagination token either. So essentially we’d need to write our own paginator just to get the continuation token.
@LeonLuttenberger @cnfait Each page here has QueryId and NextToken. We can change _rows_to_df to add attributes to the data frame it produces if any are passed and pass those attributes if chunked=True