opensearch-project
[FEATURE] Update to Apache Lucene 10 for 3.0.0
Is your feature request related to a problem? Please describe. The Apache Lucene 10 is released [1]!:
[1] https://cwiki.apache.org/confluence/display/LUCENE/Release+Notes+10.0.0
Notable features:
- https://github.com/apache/lucene/pull/13542
- https://github.com/apache/lucene/pull/12873
- https://github.com/apache/lucene/pull/13337
Describe the solution you'd like
The OpenSearch is using Apache Lucene 9.x release line for 3.x and 2.x release lines. We've never tried to integrate the Apache Lucene 10 (either as spike or to 3.0.0) but we probably should start doing that shortly.
Describe alternatives you've considered Keep Apache Lucene 9.x
Additional context Depends on https://github.com/opensearch-project/OpenSearch/issues/10745 since Apache Lucene 10 has JDK-21 baseline
I started this effort on a separate branch a long time ago shortly after finishing the multi month 9.0.0 upgrade - which was quite painful. 10.0.0 isn't as painful (since 9.0.0 had to deal w/ byte order changes) but there were JDK conflicts I need to resolve by rebasing to main. I'll take care of that and throw it on a feature branch if others want to contribute so we're not redoing the work I already did.
There's a 10.0 snapshot already but it's quite stale. Feel free to kick off a new one from lucene's main branch.
@reta @nknize can you please confirm if this change can be included in 2.x without breaking existing API? Basically can this change be added in a backward compatible manner in 2.x line?
We are evaluating if this change requires 3.0 release or can be included in 2.x line so need your inputs.
@reta @nknize can you please confirm if this change can be included in 2.x without breaking existing API? Basically can this change be added in a backward compatible manner in 2.x line?
@bbarani This is a Apache Lucene change - it will break every single plugin / extension sadly.
Another major change to explore: https://github.com/apache/lucene/pull/13337
One change I'm planning to explore: https://github.com/opensearch-project/OpenSearch/issues/17038 [Efficient doc values with skip index can be used in OS 3.0]
