storage-http: pagination with next url extracted from the page?

[matteoredaelli]

Hello

Qliksaas apis use pagination without offset params. They provide the "next" url in the response page

How could I tell drill to use "links.next.href" value for retreiving further records?

https://qlik.dev/apis/pagination-sorting-filtering

{
  "data": ["list", "of", "items"],
  "links": {
    "prev": { "href": null },
    "next": { "href": "https://example.org/api/v1/items?page=<next-cursor>" },
  }
}

[cgivre]

Take a look here: https://github.com/apache/drill/blob/master/contrib/storage-http/Pagination.md. Drill might already do what you need to do.

[matteoredaelli]

Thanks, you are right. It is in the section "Index / KeySet Pagination" : I need to add "nextPageParam" parameter but it is not clear which "method" I should set for "Index / KeySet Pagination" (it is indicated for the others options). I think "INDEX" as I see in teh code

  public enum PaginatorMethod {
    OFFSET,
    PAGE,
    INDEX,
    HEADER_INDEX
  }

Thanks Matteo