Iteration through paginated data

[hubgit]

Prerequisites

• [x] Put an X between the brackets on this line if you have done all of the following:
  • Checked this feature isn't already filed: [open features]
  • Checked the [feature request guidelines]

Summary

It would be helpful to be able to iterate through paginated data, via an async iterator.

Motivation

At the moment the user has to manually handle paginated responses by reading meta.next_token and setting it as params.pagination_token on a new request.

It would be helpful if it was possible to use for await (const item of client.get(path, params)) { … } (where item is an item from the data array in each response) to iterate through each item automatically.

Describe alternatives you've considered

Making requests in a do { … } while (params.pagination_token) loop.

[mdodge-ecgrow]

Not sure if this warrants a new issue or not. But I am trying to implement pagination to get all the 100+ replies in the conversation. I am getting the error: Error: Invalid Request: One or more parameters to your request was invalid.

Here is the code that is attempting to retrieve the replies:

const { data, meta } = await client.get("tweets/search/recent", {
      query: "conversation_id: " + id,
      "tweet.fields":
        "in_reply_to_user_id,author_id,created_at,conversation_id",
      max_results: 100,
      pagination_token: next_token,
    });

Also, that exact same code works perfect when I don't have the pagination_token parameter in there. So something is wonky with that parameter.

Also, I have verified that next_token is indeed the next_token from the previous query's meta object.

[hubgit]

@mdodge-ecgrow For some reason the v2 tweets/search/recent endpoint seems to use next_token as the parameter name instead of pagination_token.