hepdata icon indicating copy to clipboard operation
hepdata copied to clipboard

opensearch: use `search_after` parameter to paginate search results

Open GraemeWatt opened this issue 2 years ago • 1 comments

The current (v7.14) Elasticsearch docs say:

By default, you cannot use from and size to page through more than 10,000 hits. This limit is a safeguard set by the index.max_result_window index setting. If you need to page through more than 10,000 hits, use the search_after parameter instead.

See also Search After docs from Elasticsearch v7.1 that we use in production. Currently hepdata.net has 9484 publications, so this issue should be addressed before reaching 10,000.

(Issue pointed out by @michamos.)

GraemeWatt avatar Aug 16 '21 10:08 GraemeWatt