aws-sdk-pandas icon indicating copy to clipboard operation
aws-sdk-pandas copied to clipboard

Support `next_token` in paginated requests to `timestream.query()`

Open malachi-constant opened this issue 2 years ago • 2 comments

Is your idea related to a problem? Please describe. Support next_token in paginated requests to timestream.query()

Describe the solution you'd like Allow calls to timestream.query() to support paginated return of DataFrames with a next_token to allow client to manage pagination of data returned from timestream.

P.S. Please do not attach files as it's considered a security risk. Add code snippets directly in the message body as much as possible.

malachi-constant avatar Aug 31 '22 17:08 malachi-constant

It should be possible to add the continuation token into the attrs property of Pandas. Unfortunately, the docs says that the feature is experimental and could change at any time.

Another issue is that the boto3 paginator doesn’t return the pagination token either. So essentially we’d need to write our own paginator just to get the continuation token.

LeonLuttenberger avatar Sep 01 '22 21:09 LeonLuttenberger

@LeonLuttenberger @cnfait Each page here has QueryId and NextToken. We can change _rows_to_df to add attributes to the data frame it produces if any are passed and pass those attributes if chunked=True

kukushking avatar Sep 06 '22 11:09 kukushking