neural-search icon indicating copy to clipboard operation
neural-search copied to clipboard

Implement parallel execution of sub-queries for hybrid search

Open VijayanB opened this issue 1 year ago • 1 comments

Description

  1. Add new thread pool to schedule tasks that are related to hybrid query execution
  2. Register executor builders with Plugin
  3. Use Lucene's Task Executor to execute and collect results
  4. Parallelize Query re-write
  5. Parallelize score supplier creation
  6. Parallelize build hybrid scores

Issues Resolved

#279

Check List

  • [x] New functionality includes testing.
    • [x] All tests pass
  • [ ] New functionality has been documented.
    • [ ] New functionality has javadoc added
  • [x] Commits are signed as per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

VijayanB avatar May 15 '24 20:05 VijayanB

Codecov Report

Attention: Patch coverage is 83.78378% with 12 lines in your changes are missing coverage. Please review.

Please upload report for BASE (feature/parallelize-hybrid-search@806042c). Learn more about missing BASE report.

:exclamation: Current head 2750ad4 differs from pull request most recent head bc6b885

Please upload reports for the commit bc6b885 to get more accurate results.

Files Patch % Lines
...ch/neuralsearch/executors/HybridQueryExecutor.java 44.44% 4 Missing and 1 partial :warning:
...org/opensearch/neuralsearch/query/HybridQuery.java 89.47% 2 Missing :warning:
...ensearch/neuralsearch/query/HybridQueryScorer.java 86.66% 2 Missing :warning:
...ensearch/neuralsearch/query/HybridQueryWeight.java 85.71% 2 Missing :warning:
...g/opensearch/neuralsearch/plugin/NeuralSearch.java 50.00% 1 Missing :warning:
Additional details and impacted files
@@                         Coverage Diff                          @@
##             feature/parallelize-hybrid-search     #749   +/-   ##
====================================================================
  Coverage                                     ?   84.89%           
  Complexity                                   ?      812           
====================================================================
  Files                                        ?       65           
  Lines                                        ?     2490           
  Branches                                     ?      410           
====================================================================
  Hits                                         ?     2114           
  Misses                                       ?      213           
  Partials                                     ?      163           

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar May 16 '24 23:05 codecov[bot]

@navneet1v Can you take a look at one more time? Thanks

VijayanB avatar May 24 '24 02:05 VijayanB

@VijayanB I will reviewing the PR today in sometime.

vibrantvarun avatar May 24 '24 17:05 vibrantvarun