apache
Bump up Java version to 21
https://issues.apache.org/jira/browse/SOLR-17321
Bump up Java version to 21. Although we have decided to have different versions for Solr and SolrJ to 21 and 17 respectively. Currently in this PR, both of them are pointing to 21 to seek feedback and to test everything. Meanwhile looking for way to set SolrJ to 17.
This happens some time!
https://github.com/apache/solr/actions/runs/10627729009/job/29461325476?pr=2682
Task :solr:benchmark:compileJava FAILED An exception has occurred in the compiler (21.0.4). Please file a bug against the Java compiler via the Java bug reporting page (https://bugreport.java.com/) after checking the Bug Database (https://bugs.java.com/) for duplicates. Include your program, the following diagnostic, and the parameters passed to the Java compiler in your report. Thank you. java.lang.VirtualMachineError: Out of space in CodeCache for adapters at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment$DiscoveredProcessors$ProcessorStateIterator.next(JavacProcessingEnvironment.java:824) at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment.discoverAndRunProcs(JavacProcessingEnvironment.java:922) at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment$Round.run(JavacProcessingEnvironment.java:1265) at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment.doProcessing(JavacProcessingEnvironment.java:1380) at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.processAnnotations(JavaCompiler.java:1272) at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:946) at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.lambda$doCall$0(JavacTaskImpl.java:104) at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.invocationHelper(JavacTaskImpl.java:152) at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:100) at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:94) at org.gradle.internal.compiler.java.IncrementalCompileTask.call(IncrementalCompileTask.java:92) at org.gradle.api.internal.tasks.compile.AnnotationProcessingCompileTask.call(AnnotationProcessingCompileTask.java:94) at org.gradle.api.internal.tasks.compile.ResourceCleaningCompilationTask.call(ResourceCleaningCompilationTask.java:57) at org.gradle.api.internal.tasks.compile.JdkJavaCompiler.execute(JdkJavaCompiler.java:55) at org.gradle.api.internal.tasks.compile.JdkJavaCompiler.execute(JdkJavaCompiler.java:39) at org.gradle.api.internal.tasks.compile.NormalizingJavaCompiler.delegateAndHandleErrors(NormalizingJavaCompiler.java:98) at org.gradle.api.internal.tasks.compile.NormalizingJavaCompiler.execute(NormalizingJavaCompiler.java:52) at org.gradle.api.internal.tasks.compile.NormalizingJavaCompiler.execute(NormalizingJavaCompiler.java:38) at
./gradlew compileJava seem to work fine. Only If we find a way to set Jdk to 21 for all the test packages. Because currently, test cases from api not able to find classes from other projects that are on higher version, and that's where it's failing.
Note that "api" module should also be Java Language 17, as it's included with SolrJ.
Looks like below config is working. It sets solrj and related projects to Jdk17 and rest of the projects to Jdk21 for main. And, for all the test packages they are using Jdk21.
javac.gradle
// Use 'release' flag instead of 'source' and 'target'
tasks.withType(JavaCompile) {
compileTestJava {
options.compilerArgs += ["--release", JavaVersion.toVersion(scriptDepVersions['min-java-version']).toString()]
}
compileJava {
options.compilerArgs += ["--release", project.minJavaVersion.toString()]
}
}
Now, to test it. I unpack the builds and verify it in MANIFEST.MF.
solrj
Specification-Title: Apache Solr Search Server: solrj
X-Compile-Source-JDK: 17
X-Compile-Target-JDK: 17
X-Build-JDK: 21.0.3 (Oracle Corporation 21.0.3+7-LTS-152)
X-Build-OS: Mac OS X aarch64 14.3
api
Specification-Title: Apache Solr Search Server: api
X-Compile-Source-JDK: 17
X-Compile-Target-JDK: 17
X-Build-JDK: 21.0.3 (Oracle Corporation 21.0.3+7-LTS-152)
X-Build-OS: Mac OS X aarch64 14.3
solr-zookeeper
Specification-Title: Apache Solr Search Server: solrj-zookeeper
X-Compile-Source-JDK: 17
X-Compile-Target-JDK: 17
X-Build-JDK: 21.0.3 (Oracle Corporation 21.0.3+7-LTS-152)
X-Build-OS: Mac OS X aarch64 14.3
solrj-streaming
Specification-Title: Apache Solr Search Server: solrj-streaming
X-Compile-Source-JDK: 17
X-Compile-Target-JDK: 17
X-Build-JDK: 21.0.3 (Oracle Corporation 21.0.3+7-LTS-152)
X-Build-OS: Mac OS X aarch64 14.3
core
Specification-Title: Apache Solr Search Server: core
X-Compile-Source-JDK: 21
X-Compile-Target-JDK: 21
X-Build-JDK: 21.0.3 (Oracle Corporation 21.0.3+7-LTS-152)
X-Build-OS: Mac OS X aarch64 14.3
But I would really appreciate If someone else test it and check If we able to achieve the different version for solr (Jdk21) and solrj(jdk17).
Nice work. I toyed with it a bit, including tinkering with toolchains. Avoiding toolchains avoids IntelliJ's Gradle's check to tell you that we really should be using a newer Gradle Version:
Your build is currently configured to use incompatible Java 21.0.4 and Gradle 8.4. Cannot sync the project.
We recommend upgrading to Gradle version 8.8.
The minimum compatible Gradle version is 8.5.
The maximum compatible Gradle JVM version is 20.
Possible solutions:
- Upgrade to Gradle 8.8 and re-sync
- Upgrade to Gradle 8.5 and re-sync
- Use Java 20 as Gradle JVM: Open Gradle settings
Why aren't we using toolchains?: https://docs.gradle.org/current/userguide/toolchains.html
The Docker build.gradle needs to refer to a JDK 21 base image, not JDK 17.
BTW, when I checked out your branch and before I tinkered with it, I simply tried to get IntelliJ to sync it and then view IntelliJ's module language versions to see if it would get confused on the SolrJ modules, given differing desired language levels for tests. It was confused; it thought the whole module should use JDK 17, even the test sourcepath. I suspect the issue is that we don't use proper metadata (i.e. toolchain) to convey to IntelliJ what's going on.
The toolchain cannot be set at the JavaCompile level. Because we are using different Jdk for main and test package. To set at project level, below is the config. But this fails!
allprojects {
plugins.withType(JavaPlugin) {
java {
toolchain {
languageVersion = JavaLanguageVersion.of(17)
}
}
}
}
Next, we have different toolchain for the tasks. And below is the way to configure different toolchain for compileJava and compileTestJava.
tasks.withType(JavaCompile) {
compileTestJava {
javaCompiler = javaToolchains.compilerFor {
languageVersion = JavaLanguageVersion.of(JavaVersion.toVersion(scriptDepVersions['min-java-version']).toString())
}
options.compilerArgs += ["--release", JavaVersion.toVersion(scriptDepVersions['min-java-version']).toString()]
}
compileJava {
javaCompiler = javaToolchains.compilerFor {
languageVersion = JavaLanguageVersion.of(project.minJavaVersion.toString())
}
options.compilerArgs += ["--release", project.minJavaVersion.toString()]
}
}
This works! I ran with ./gradlew :solr:api:build --info and I can see different toolchain being loaded
Task ':solr:api:compileJava' is not up-to-date because:
Output property 'destinationDirectory' file /Users/sanjaydutt/Documents/solr/solr/api/build/classes/java/main has been removed.
Output property 'destinationDirectory' file /Users/sanjaydutt/Documents/solr/solr/api/build/classes/java/main/org has been removed.
Output property 'destinationDirectory' file /Users/sanjaydutt/Documents/solr/solr/api/build/classes/java/main/org/apache has been removed.
Output property 'options.generatedSourceOutputDirectory' file /Users/sanjaydutt/Documents/solr/solr/api/build/generated/sources/annotationProcessor/java/main has been re
moved.
The input changes require a full rebuild for incremental task ':solr:api:compileJava'.
Full recompilation is required because no incremental change information is available. This is usually caused by clean builds or changing compiler arguments.
Compiling with toolchain '/Library/Java/JavaVirtualMachines/jdk-17.jdk/Contents/Home'.
Compiling with JDK Java compiler API.
Class dependency analysis for incremental compilation took 0.003 secs.
Created classpath snapshot for incremental compilation in 0.0 secs.
work action resolve main (project :solr:api) (Thread[#760,Execution worker Thread 3,5,main]) started.
Resolve mutations for :solr:api:processResources (Thread[#760,Execution worker Thread 3,5,main]) started.
> Task :solr:api:compileTestJava
Caching disabled for task ':solr:api:compileTestJava' because:
Build cache is disabled
Task ':solr:api:compileTestJava' is not up-to-date because:
Task has failed previously.
The input changes require a full rebuild for incremental task ':solr:api:compileTestJava'.
Full recompilation is required because no incremental change information is available. This is usually caused by clean builds or changing compiler arguments.
Compiling with toolchain '/Library/Java/JavaVirtualMachines/jdk-21.jdk/Contents/Home'.
Compiling with JDK Java compiler API.
Class dependency analysis for incremental compilation took 0.0 secs.
Created classpath snapshot for incremental compilation in 0.003 secs.
Resolve mutations for :solr:api:analyzeTestClassesDependencies (Thread[#757,included builds,5,main]) started.
However, jar task is still, I think ,running with whatever java_home is
Because whenever I open MANIFEST.MF, Build-JDK is 21. I mean we are successfully able to compile files using jdk 17, however whenever jar task is running it's running via 21.
Specification-Title: Apache Solr Search Server: api
X-Compile-Source-JDK: 17
X-Compile-Target-JDK: 17
X-Build-JDK: (21.0.3)
X-Build-OS: Mac OS X aarch64 14.3
I try to add the toolchain config to jar-manifest.gradle, but it's not working
doFirst {
def toolchain = javaToolchains.compilerFor {
languageVersion = JavaLanguageVersion.of(project.minJavaVersion.toString())
}
}
manifest {
attributes([
"Extension-Name" : implementationTitle,
"Implementation-Vendor" : "The Apache Software Foundation",
"Implementation-Title" : implementationTitle,
"Implementation-Version": implementationVersion,
"Specification-Vendor" : "The Apache Software Foundation",
"Specification-Version" : project.baseVersion,
"Specification-Title" : title,
// Evaluate these properties lazily so that the defaults are applied properly.
"X-Compile-Source-JDK" : "${-> project.sourceCompatibility}",
"X-Compile-Target-JDK" : "${-> project.targetCompatibility}",
"X-Build-JDK" : "${System.properties['java.version']} (${System.properties['java.vendor']} ${System.properties['java.vm.version']})",
"X-Build-OS" : "${System.properties['os.name']} ${System.properties['os.arch']} ${System.properties['os.version']}"
])
}
When it comes to Intellij module language level, I am not sure How we will solve that? Given that all solrj and realted modules have two jdk versions now.
We use Toolchain to enforce the specific version of the tools we want to use. Here are some use cases:
- If our system's JDK is set to 21, but I want to run a project on a different Java version.
- If I need a specific Java version from a vendor for all tasks.
- If I want to run Gradle and the project on different Java versions.
In our scenario, if the system is configured with JDK 21, that version will be used to run all tasks for Solr and SolrJ. JDK 21 can generate class files with version 61 without issues. However, if we want SolrJ to compile only with JDK 17, Toolchain can accomplish that. Otherwise, the Java version used by Gradle will be applied globally.
From Intellij docs:
https://www.jetbrains.com/help/idea/gradle.html#gradle_jvm
Language level: the language level settings are applied for a source root or for a module. If a Gradle project has a single linked project then the project default language level is set to the minimum language level among the module language levels. The module language level is set to sourceCompatibility in the build.gradle file.
IMO, There is no way to configure toolchain for jar task like we can do for JavaCompile. Even the JarTask has no mention of compiler like JavaCompiler has.
https://github.com/gradle/gradle/blob/master/platforms/jvm/platform-jvm/src/main/java/org/gradle/jvm/tasks/Jar.java https://github.com/gradle/gradle/blob/master/platforms/jvm/language-java/src/main/java/org/gradle/api/tasks/compile/JavaCompile.java#L202
My question is that does it even matter which jdk version actually package the class files? Given that we made sure that compileJava task ran only by appropriate jdk.
However, jar task is still, I think ,running with whatever java_home is
It's normal & acceptable that the entire build itself do JAR packaging etc with the same running JVM, that which Gradle is using -- JDK 21. We shouldn't try to change that or validate the manifest entry for the build JDK. It's just some FYI metadata.
RE IntelliJ -- lots of us use IntelliJ; if IntelliJ will not tolerate Java 21 language features in a SolrJ test, we'll have to use Java 17 for those tests. Okay, not a big deal.
gradle Auto-provisioning is disabled.
./gradlew -q javaToolchains
+ Options
| Auto-detection: Enabled
| Auto-download: Disabled
Build is failing due to that. It's like now we are basically forcing the version of tools that being used, in this case two Jdk version 17 and 21. Otherwise, If you have both version available then no issues.
> Task :solr:api:compileJava FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':solr:api:compileJava'.
> Error while evaluating property 'javaCompiler' of task ':solr:api:compileJava'.
> Failed to calculate the value of task ':solr:api:compileJava' property 'javaCompiler'.
> No matching toolchains found for requested specification: {languageVersion=17, vendor=any, implementation=vendor-specific} for LINUX on x86_64.
> No locally installed toolchains match and toolchain download repositories have not been configured.
There is a way to enable it via gradle.properties. (I have to check how I can make this file change on crave, it's not picking it up.)
# Disable auto JVM provisioning (we don't use toolchains yet but want no surprises).
org.gradle.java.installations.auto-download=false
There is somewhat related discussion regarding the same in alternative-jdk-support.gradle.
https://github.com/apache/solr/blob/a4cc5809c52eceffae10cb219299f40e0755e9a9/gradle/testing/alternative-jdk-support.gradle#L25-L35
I might have misunderstood Toolchains; if it means having additional JDKs around then I don't think we want to require that. It's sufficient to have only JDK 21 and configure the language level for certain parts.
As I mentioned, IntelliJ isn't recognizing the language distinction between tests vs non-tests so I recommend simply using the same language per module. Other than that, I suppose it's ready to merge.
At some point (could be now), we should review all the extra options we pass to java when starting Solr to see if they are still sensible. Some may be obsolete.
If I set sourceCompatibility at the same level as "--release" flag then It worked, which is basically set it the package level (main or test) instead of Project. Now after this, We can use JDK 21 for all test packages.
Documentation from Intellij is very much relevant at package level as well. For language level it looks at sourceCompatibility.
Language level: the language level settings are applied for a source root or for a module. If a Gradle project has a single linked project then the project default language level is set to the minimum language level among the module language levels. The module language level is set to sourceCompatibility in the build.gradle file.
We may need to remove some of the compiler options as well. As they may not be required anymore with the upgrade, also may be two different sets of compiler options -- one for JDK 17 and another JDK 21. After upgrading to 17, Lucene removed some of the flags and added new ones https://github.com/apache/lucene/pull/628
JDK 14 Release Notes -XX:UseConcMarkSweepGC has been removed. I believe, we still using in solr.cmd.in.
Other than that I have gone through the release notes and did'nt find anything which we are still using and has been removed.
EDIT: I found -XX:+UseG1GC in both solr.cmd and solr. Also, saw in action while starting it on OS X.
I think that was just an old reference to CMS in solr.cmd that we are not using anymore. (Actually UseConcMarkSweepGC was commented in both scripts. So we can ignore it!)
Since JDK 14 https://openjdk.org/jeps/363 Concurrent Mark Sweep (CMS) garbage collector is removed. So may be can remove that and whatever is related to it (Which I am not fully aware of)
Next:
- Change the smokeTestRelease. One of the change that I can think of is to test Jar Java version for SolrJ and Solr. (Separate PR)
- Whenever we plan to merge it, we need to make the Java version change on CI servers. I wonder Do we even need that new file which we added for crave; instead, we can change Java version on crave as well.
Trying to suppress security warnings (excessive logging) but did not work:
WARNING: A command line option has enabled the Security Manager
WARNING: The Security Manager is deprecated and will be removed in a future release
Sadly the PR diff is now confusing; seems to have dependency updates and stuff from main but GitHub isn't detecting/realizing that this PR shouldn't be bringing such changes if merged. Or you inadvertently did stuff here?
Security Manager: decouple from this PR; a dev list thread if we want to explore disabling by default or options to suppress this. Jenkins: Yes, I'll updated some but didn't update all. Hopefully you can continue that; I find myself too busy.
You didn't update bin/solr options yet but you plan to remove at least one obsolete option. Further JDK tuning can be a new dev list thread.
Whenever we plan to merge it, we need to make the Java version change on CI servers. I wonder Do we even need that new file which we added for crave; instead, we can change Java version on crave as well.
CC @uvatbc
Sadly the PR diff is now confusing; seems to have dependency updates and stuff from main but GitHub isn't detecting/realizing that this PR shouldn't be bringing such changes if merged. Or you inadvertently did stuff here?
umm looks like it's a common issue. https://stackoverflow.com/questions/16306012/github-pull-request-showing-commits-that-are-already-in-target-branch I tried one of the solution -- changing base but did'nt worked. I think I will just create another PR, If that's alright?
No please don't create yet another PR just because the diff isn't looking clear. I've already reviewed everything so I recommend a rebase & force push.
You're super close to merging this.
- remove
UseConcMarkSweepGCthat is obsolete - CHANGES.txt add an entry for Solr 10 "Other Changes"
-
major-changes-in-solr-10.adocshould be updated. Add a Misc section maybe? And in the SolrJ section, add a bullet for Java 17 dependency.
-XX:-UseBiasedLocking removed in JDK 21. And, due to which Benchmarking is failing. Will be removing it.