Lucene 9.3.0

[tarzanek]

continuation of https://github.com/oracle/opengrok/pull/3833

[tarzanek]

so jar and class analyzer are still broken after upgrade

[tarzanek]

but why the javadoc fails is a mystery for me also mac build wasn't able to download some cve @vladak any clues about above?

[vladak]

but why the javadoc fails is a mystery for me also mac build wasn't able to download some cve @vladak any clues about above?

the CVE download problem is long standing in the macos environment and can be ignored.

[vladak]

As for the Javadoc errors, I can see these in the log:

/home/runner/work/opengrok/opengrok/opengrok-indexer/src/main/java/org/opengrok/indexer/search/context/OGKUnifiedHighlighter.java:72: error: @param name not found
Error:       * @param indexSearcher a required instance
Error:                ^
Error:  /home/runner/work/opengrok/opengrok/opengrok-indexer/src/main/java/org/opengrok/indexer/search/context/OGKUnifiedHighlighter.java:73: error: @param name not found
Error:       * @param indexAnalyzer a required instance
Error:                ^

Looking at the changes, this is valid error. Plus the missing new param.

[tarzanek]

thnx, fixed javadocs, now I need to fix the jar and class analyzers, I have no clue why they got broken perhaps https://github.com/oracle/opengrok/pull/3912/commits/1e1dc7c1b7cd570c22bafe59b05adb8c315506fd ? ... hmm, no, this doesn't seem to be the problem, need to debug those analyzers

[vladak]

rebase ?

[tarzanek]

need rebase ... rebased with https://github.com/oracle/opengrok/commit/04d76760745074687f6be81f7b75a193ea5f3095

[tarzanek]

see https://github.com/oracle/opengrok/pull/4100#issuecomment-1411045335

[tarzanek]

actually above is relevant for jdk > jdk 11 for jdk 11 we see: https://github.com/apache/lucene/pull/11/files java.lang.IllegalArgumentException: Inconsistency of field data structures across documents for field [defs] of doc [1]. store term vector: expected 'true', but it has 'false'. for the jar file

[tarzanek]

reindex is definitely needed I fixed a bug in doc schema (for indexed jar and class files) also format of index changed (but because of above auto upgrade won't work)

[tarzanek]

@vladak the windows tests are sloppy and when I switched to other than jdk11 I saw weird concurrency errors so I guess we will need to do some sanity checks on parallelism in OpenGrok since JDK 17 and JDK 19 definitely behave in weird way with it (ev. lucene has bugs with jdk 17 and jdk 19, it's hard for me to decide)

[tarzanek]

can you rerun the windows test @vladak ? (it seems like some jersey weirdness/race on windows, every x runs it reproduces)

[tarzanek]

@ahornace any clues why those SuggesterControllerTest tests are not stable on windows ?

[vladak]

@ahornace any clues why those SuggesterControllerTest tests are not stable on windows ?

This is #4108. I dug somewhat deep I think however haven't been able to root cause it yet.

[vladak]

can you rerun the windows test @vladak ? (it seems like some jersey weirdness/race on windows, every x runs it reproduces)

Kicked off rerun. It fails more often than not however.

[tarzanek]

ok, failed again, so your call @vladak from my side this PR is ready for merge

[ahornace]

any clues why those SuggesterControllerTest tests are not stable on windows ?

I'm sorry, I have no idea. I agree that it shouldn't be a blocker for this change. Maybe we could disable them for Windows for now?

I don't have access to a Windows station for reproducing the issue. I will try to look into it though.

[tarzanek]

at least the tests are stable on linux :-) (now I am definitely done with this PR)

[vladak]

any clues why those SuggesterControllerTest tests are not stable on windows ?

I'm sorry, I have no idea. I agree that it shouldn't be a blocker for this change. Maybe we could disable them for Windows for now?

I don't have access to a Windows station for reproducing the issue. I will try to look into it though.

it happens on macos, although not as frequently and I was able to reproduce it on linux even when single stepping in the debugger.

[vladak]

is reindex from scratch needed ?

[tarzanek]

is reindex from scratch needed ?

yes, unfortunately there was a bug in schema (the jar/java analyzers saved defs into wrong type and previous lucene didn't check for this, now all analyzers have to use proper type defined in plain/file analyzers and later analyzers cannot override it - https://github.com/oracle/opengrok/pull/3912/commits/984b420512728c1a2a156aecf126cf928d089de6)

• this is major upgrade

[vladak]

Thanks !