openverse-api icon indicating copy to clipboard operation
openverse-api copied to clipboard

Start from previous window end in `_post_process_results`

Open AetherUnbound opened this issue 3 years ago • 0 comments

Description

Presently, the _post_process_results function iterates through the results & filters out dead links, querying Elasticsearch when more results are necessary. While the end of the window is increased on subsequent searches, the searches themselves when called recursively within the function always start from the initial position. This means that subsequent queries become larger and larger even though the same results are being returned and processed.

Once #873 is complete, we should modify this function so that the start value is moved to the previous end in subsequent calls. This should reduce the range of data that's queried against Elasticsearch.

Implementation

  • [ ] 🙋 I would be interested in implementing this feature.

AetherUnbound avatar Aug 10 '22 20:08 AetherUnbound