hepdata
hepdata copied to clipboard
opensearch: use `search_after` parameter to paginate search results
The current (v7.14) Elasticsearch docs say:
By default, you cannot use
from
andsize
to page through more than 10,000 hits. This limit is a safeguard set by theindex.max_result_window
index setting. If you need to page through more than 10,000 hits, use thesearch_after
parameter instead.
See also Search After docs from Elasticsearch v7.1 that we use in production. Currently hepdata.net has 9484 publications, so this issue should be addressed before reaching 10,000.
(Issue pointed out by @michamos.)