Paginate until no records or limit

Open ferrumboll opened this issue 1 year ago • 3 comments

Hello, There are some APIs that don't have a next page identifier in the response, the only way is to keep querying until a manual limit or until the API gives 0 items.

Is this possible with this Tap?

Jan 17 '24 21:01 ferrumboll

I could try to do a PR with an extra Paginator for this, if it is not possible right now.

Jan 17 '24 22:01 ferrumboll

I do not believe that is supported, so if you are able to add a PR, that would be the quickest path to adding support for it.

Jan 18 '24 15:01 jlloyd-widen

fwiw starting with 0.35.0 (see https://github.com/meltano/sdk/pull/1918) the sdk will exit the pagination loop if the page contains no records :)

Mar 15 '24 15:03 edgarrmondragon