unipop
unipop copied to clipboard
Elasticsearch Scroll API
Use elasticsearch query Scroll API to iterate many results in the StartStep. https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html
According to the docs "Scrolling is not intended for real time user requests, but rather for processing large amounts of data, e.g. in order to reindex the contents of one index into a new index with a different configuration". Should we use Scroll in Unipop? If so, should we always use it? maybe only in StartStep (Assuming most times it will iterate a large amount of results).
The scroll functionality is already implemented in QueryIterator
, but its currently unused.
need to implement #103 first