Pager data from the response

[kakoni]

Hi and thank you for this wonderful plugin.

Quite often the actual http response (for instance json) contains information about the paging (current page, total pages etc). Do you see any possibility to use that information together with pager? My current solution is to do two two queries, first one using curl+jq to get information about total pages and then pass this information to embulk as environment variable.

[takumakanari]

Hi @kakoni, thanks for using my plugin.

Currently, it's difficult to handle pagination by looking responses because the plugin does not parse response text into any format.

As you say, there is some solution to handle pagination, just like using curl + jq before using this plugin.