openlibrary
openlibrary copied to clipboard
Fix: Improve Solr query stability for languages and subjects
Partially Closes #11546
[Fix]
Technical
Changes:
- openlibrary/plugins/worksearch/languages.py: Set
_pass_time_allowed=Trueto the Solr query inget_all_language_counts, this makes Solr return the partial results it has collected so far instead of crashing. - openlibrary/plugins/worksearch/subjects.py: Changed the
publish_yearfacet limit from -1 to 2000 inSubjectEngine, this makes sure Solr returns the top 2000 most common years which covers up all legitimate historical years while filtering out dirty data like typos or invalid years which would cause performance issues
Testing
Ran automated tests (openlibrary/plugins/worksearch/tests/test_worksearch.py)
Created a custom script (openlibrary/plugins/worksearch/tests/test_verification.py) to mock solr and verify that the new parameters are correctly passed
- Timeout handling: Simulated a timeout returning
partialResults: Trueand confirmed thatget_all_language_countsreturns the available data instead of raising an exception. - Parameter Verification: Confirmed that
_pass_time_allowed=Trueis explicitly passed in the Solr query parameters. - Facet Limit: Confirmed that the
publish_yearfacet request includeslimit: 2000.
Screenshot
Stakeholders
I've been trying to get the python_tests to pass but my test_verification.py file makes it fail