OpenSearch icon indicating copy to clipboard operation
OpenSearch copied to clipboard

Published 20 hours ago • verified publisher icon opensearch-project

Bump antlr4 from 4.9.3 to 4.11.1

Open reta opened this issue 3 years ago • 4 comments

Signed-off-by: Andriy Redko [email protected]

Description

Manually bumping antlr4 version and regenerate parsers

Issues Resolved

Closes https://github.com/opensearch-project/OpenSearch/pull/4412 and https://github.com/opensearch-project/OpenSearch/pull/4412

Check List

  • [ ] New functionality includes testing.
    • [ ] All tests pass
  • [ ] New functionality has been documented.
    • [ ] New functionality has javadoc added
  • [X] Commits are signed per the DCO using --signoff
  • [ ] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

reta avatar Sep 19 '22 13:09 reta

Gradle Check (Jenkins) Run Completed with:

  • RESULT: FAILURE :x:
  • URL: https://build.ci.opensearch.org/job/gradle-check/3199/
  • CommitID: c4220a125c00584bb817ec8030ef1e73a2fb9464

github-actions[bot] avatar Sep 19 '22 13:09 github-actions[bot]

Gradle Check (Jenkins) Run Completed with:

  • RESULT: FAILURE :x:
  • URL: https://build.ci.opensearch.org/job/gradle-check/3200/
  • CommitID: 5081a5682ca1caed826bdd355c6b9db009c241ef

github-actions[bot] avatar Sep 19 '22 13:09 github-actions[bot]

Gradle Check (Jenkins) Run Completed with:

  • RESULT: FAILURE :x:
  • URL: https://build.ci.opensearch.org/job/gradle-check/3207/
  • CommitID: 5081a5682ca1caed826bdd355c6b9db009c241ef

github-actions[bot] avatar Sep 19 '22 13:09 github-actions[bot]

Ah, bad news, the Apache Lucene still uses some Antlr 3.x bits (serialized ATN, see here [1] please), but the compatibility with Antlr 3.x was removed [2] recently. As such, Apache Lucene is not compatible with Antlr 4.10+ :(

[1] https://github.com/apache/lucene/blob/main/lucene/expressions/src/java/org/apache/lucene/expressions/js/JavascriptLexer.java#L189 [2] https://github.com/antlr/antlr4/commit/c68e127a7cf14470565d6e6ae1eff06db3e56ea7

reta avatar Sep 19 '22 14:09 reta

Gradle Check (Jenkins) Run Completed with:

  • RESULT: FAILURE :x:
  • URL: https://build.ci.opensearch.org/job/gradle-check/7867/
  • CommitID: 8397dd7c28304122f7e6102e952fde4494ae6e8f Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions[bot] avatar Dec 09 '22 21:12 github-actions[bot]

Getting there :) https://github.com/apache/lucene/issues/11788 is merged, :crossed_fingers: it will be backported to 9.5

reta avatar Dec 16 '22 13:12 reta

Gradle Check (Jenkins) Run Completed with:

  • RESULT: SUCCESS :white_check_mark:
  • URL: https://build.ci.opensearch.org/job/gradle-check/8191/
  • CommitID: e5fdf9d0f16a6b220620837aa5b685641cc0ad08

github-actions[bot] avatar Dec 16 '22 16:12 github-actions[bot]

Codecov Report

Merging #4546 (e5fdf9d) into main (cb26035) will decrease coverage by 0.04%. The diff coverage is 10.52%.

@@             Coverage Diff              @@
##               main    #4546      +/-   ##
============================================
- Coverage     70.95%   70.91%   -0.05%     
+ Complexity    58304    58296       -8     
============================================
  Files          4733     4733              
  Lines        278256   278256              
  Branches      40249    40249              
============================================
- Hits         197441   197319     -122     
- Misses        64628    64771     +143     
+ Partials      16187    16166      -21
Impacted Files Coverage Δ
...arch/painless/antlr/PainlessParserBaseVisitor.java 1.26% <ø> (ø)
.../org/opensearch/painless/antlr/PainlessParser.java 68.43% <5.55%> (ø)
...a/org/opensearch/painless/antlr/PainlessLexer.java 68.18% <100.00%> (ø)
...adonly/AddIndexBlockClusterStateUpdateRequest.java 0.00% <0.00%> (-75.00%) :arrow_down:
...readonly/TransportVerifyShardIndexBlockAction.java 9.75% <0.00%> (-73.18%) :arrow_down:
.../index/shard/IndexShardNotRecoveringException.java 0.00% <0.00%> (-50.00%) :arrow_down:
.../opensearch/test/store/MockFSDirectoryFactory.java 22.66% <0.00%> (-46.67%) :arrow_down:
...rch/client/transport/NoNodeAvailableException.java 28.57% <0.00%> (-42.86%) :arrow_down:
...h/action/ingest/SimulateDocumentVerboseResult.java 60.71% <0.00%> (-39.29%) :arrow_down:
...indices/readonly/TransportAddIndexBlockAction.java 20.68% <0.00%> (-37.94%) :arrow_down:
... and 476 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

codecov-commenter avatar Dec 16 '22 16:12 codecov-commenter

@dblock @nknize I think we are good to go! (no backport since hard dependency on Apache Lucene 9.5), thanks guys!

reta avatar Dec 16 '22 16:12 reta

I know this is merged, but it seems to be causing fatal errors on the darwin-arm64-tar builds when running on m1 macs.

» ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [runTask-0] fatal error in thread [main], exiting
»  java.lang.NoClassDefFoundError: org/opensearch/index/rankeval/RankEvalAction
»       at java.lang.Class.getDeclaredConstructors0(Native Method) ~[?:?]
»       at java.lang.Class.privateGetDeclaredConstructors(Class.java:3373) ~[?:?]
»       at java.lang.Class.getConstructors(Class.java:2060) ~[?:?]
»       at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:765) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:731) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:533) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:195) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.node.Node.<init>(Node.java:427) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.node.Node.<init>(Node.java:354) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»       at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  Caused by: java.lang.ClassNotFoundException: org.opensearch.index.rankeval.RankEvalAction
»       at java.net.URLClassLoader.findClass(URLClassLoader.java:445) ~[?:?]
»       at java.lang.ClassLoader.loadClass(ClassLoader.java:587) ~[?:?]
»       at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:872) ~[?:?]
»       at java.lang.ClassLoader.loadClass(ClassLoader.java:520) ~[?:?]
»       ... 19 more

When I checkout the commit prior git checkout ff43d768b10f97bac35a2638c2c6bc2b642cd10f then my ./gradlew run runs fine.

acarbonetto avatar Dec 22 '22 00:12 acarbonetto

When I checkout the commit prior git checkout ff43d768b10f97bac35a2638c2c6bc2b642cd10f then my ./gradlew run runs fine.

I have difficulties understanding how this specific change impacts rank-eval plugin (especially on M1), @acarbonetto do you build the complete distribution locally?

reta avatar Dec 22 '22 00:12 reta

I don't understand it either. I've been trying to work through the changes and I don't see the relevance.

do you build the complete distribution locally?

Yes. I'm using :distribution:archives:darwin-arm64-tar:assemble and :distribution:archives:darwin-arm64-tar:build to build the install and running it from ./distribution/archives/darwin-arm64-tar/build/install/opensearch-3.0.0-SNAPSHOT/bin/opensearch

acarbonetto avatar Dec 22 '22 17:12 acarbonetto

Workaround is to work with Java 11.

If I build with Java 17 I get the above error, but not with Java 11. Something is missing in java 17 on arm64. The intel Mac seem to compile and build fine.

acarbonetto avatar Dec 22 '22 21:12 acarbonetto

Workaround is to work with Java 11.

If I build with Java 17 I get the above error, but not with Java 11. Something is missing in java 17 on arm64. The intel Mac seem to compile and build fine.

Ah ... this is very interesting, I sadly don't have M1 at my disposal but I will try to troubleshoot the issue on first occasion. What JDK distribution you use on M1?

reta avatar Dec 22 '22 22:12 reta

What JDK distribution you use on M1?

I'm using Corretto-17.0.3.6.1.

If I found out more information, I'll let you know.

acarbonetto avatar Dec 27 '22 06:12 acarbonetto