sparrow
sparrow copied to clipboard
sparrowwallet
SystemModulesPlugin sometimes breaks reproducibility by encoding different ModuleDescriptor hashCodes
The SystemModulesPlugin used by jlink builds two class files, SystemModules$all.class and SystemModules$default.class. These class files encode and build the modules for the application. The build method takes an int as input, which is calculated by the SystemModulesPlugin as the result of ModuleDescriptor.hashCode(). The value of this hashcode is cached, and in some cases (it appears) set by a constructor.
These values can however differ when building on different machines, affecting build reproducibility. The following diffoscope output documents differences in 6 modules across the two classes for the same source built on different machines (both amd64/x86_64 Debian-based OS). It is not clear however why they differ, since ModuleDescriptor.hashCode() should be deterministic given the same module configuration.
diffoscope ~/sparrow/modules/build/modules-extracted/java.base/jdk/internal/module/SystemModules\$all.class ~/sparrow/modules/github-binaries/modules-extracted/java.base/jdk/internal/module/SystemModules\$all.class
builder87.opens(new ModuleDescriptor.Opens[0]);
builder87.uses(of20);
builder87.provides(new ModuleDescriptor.Provides[0]);
builder87.packages(Set.of("com.github.arteam.simplejsonrpc.client", "com.github.arteam.simplejsonrpc.client.builder", "com.github.arteam.simplejsonrpc.client.exception", "com.github.arteam.simplejsonrpc.client.generator", "com.github.arteam.simplejsonrpc.client.metadata"));
builder87.version("1.0");
- array[72] = builder87.build(1494625572);
+ array[72] = builder87.build(1683523855);
final Builder builder88 = new Builder("simple.json.rpc.server");
builder88.open(true);
builder88.requires(new ModuleDescriptor.Requires[] { Builder.newRequires(of12, "com.fasterxml.jackson.databind"), Builder.newRequires(of12, "com.google.common"), Builder.newRequires(of12, "java.base"), Builder.newRequires(of12, "org.slf4j"), Builder.newRequires(of12, "simple.json.rpc.core") });
builder88.exports(new ModuleDescriptor.Exports[] { Builder.newExports(of, "com.github.arteam.simplejsonrpc.server") });
builder88.opens(new ModuleDescriptor.Opens[0]);
builder88.uses(of20);
builder88.provides(new ModuleDescriptor.Provides[0]);
builder88.packages(Set.of("com.github.arteam.simplejsonrpc.server", "com.github.arteam.simplejsonrpc.server.metadata"));
builder88.version("1.0");
- array[73] = builder88.build(555683016);
+ array[73] = builder88.build(744581299);
final Builder builder89 = new Builder("tornadofx.controls");
builder89.open(true);
builder89.requires(new ModuleDescriptor.Requires[] { Builder.newRequires(of12, "java.base"), Builder.newRequires(of12, "javafx.controls") });
builder89.exports(new ModuleDescriptor.Exports[] { Builder.newExports(of, "tornadofx.control") });
builder89.opens(new ModuleDescriptor.Opens[0]);
builder89.uses(of20);
builder89.provides(new ModuleDescriptor.Provides[0]);
builder89.packages(Set.of("tornadofx.control", "tornadofx.control.skin", "tornadofx.converter", "tornadofx.property", "tornadofx.table"));
builder89.version("1.0.4");
- array[74] = builder89.build(1570093334);
+ array[74] = builder89.build(1758991617);
final Builder builder90 = new Builder("com.sparrowwallet.sparrow");
builder90.open(true);
diffoscope ~/sparrow/modules/build/modules-extracted/java.base/jdk/internal/module/SystemModules\$default.class ~/sparrow/modules/github-binaries/modules-extracted/java.base/jdk/internal/module/SystemModules\$default.class
--- /home/nyxnor/sparrow/modules/build/modules-extracted/java.base/jdk/internal/module/SystemModules$default.class
+++ /home/nyxnor/sparrow/modules/github-binaries/modules-extracted/java.base/jdk/internal/module/SystemModules$default.class
├── procyon -ec {}
@@ -449,55 +449,55 @@
builder20.requires(new ModuleDescriptor.Requires[] { Builder.newRequires(of12, "java.base"), Builder.newRequires(of12, "javafx.base"), Builder.newRequires(of12, "javafx.controls"), Builder.newRequires(of12, "javafx.graphics") });
builder20.exports(new ModuleDescriptor.Exports[] { Builder.newExports(of, "de.codecentric.centerdevice") });
builder20.opens(new ModuleDescriptor.Opens[0]);
builder20.uses(of20);
builder20.provides(new ModuleDescriptor.Provides[0]);
builder20.packages(Set.of("de.codecentric.centerdevice", "de.codecentric.centerdevice.dialogs.about", "de.codecentric.centerdevice.glass", "de.codecentric.centerdevice.icns", "de.codecentric.centerdevice.labels", "de.codecentric.centerdevice.listener", "de.codecentric.centerdevice.util"));
builder20.version("2.1.7");
- array[9] = builder20.build(-218088662);
+ array[9] = builder20.build(-29190379);
final Builder builder21 = new Builder("co.nstant.in.cbor");
builder21.open(true);
builder21.requires(new ModuleDescriptor.Requires[] { Builder.newRequires(of12, "java.base") });
builder21.exports(new ModuleDescriptor.Exports[] { Builder.newExports(of, "co.nstant.in.cbor") });
builder21.opens(new ModuleDescriptor.Opens[0]);
builder21.uses(of20);
builder21.provides(new ModuleDescriptor.Provides[0]);
builder21.packages(Set.of("co.nstant.in.cbor", "co.nstant.in.cbor.builder", "co.nstant.in.cbor.decoder", "co.nstant.in.cbor.encoder", "co.nstant.in.cbor.model"));
builder21.version("0.9");
- array[10] = builder21.build(-1386751763);
+ array[10] = builder21.build(-1197853480);
final Builder builder22 = new Builder("com.beust.jcommander");
builder22.open(true);
builder22.requires(new ModuleDescriptor.Requires[] { Builder.newRequires(of12, "java.base") });
builder22.exports(new ModuleDescriptor.Exports[] { Builder.newExports(of, "com.beust.jcommander") });
builder22.opens(new ModuleDescriptor.Opens[0]);
builder22.uses(of20);
builder22.provides(new ModuleDescriptor.Provides[0]);
builder22.packages(Set.of("com.beust.jcommander", "com.beust.jcommander.converters", "com.beust.jcommander.defaultprovider", "com.beust.jcommander.internal", "com.beust.jcommander.validators"));
builder22.version("1.81");
- array[11] = builder22.build(121114450);
+ array[11] = builder22.build(310012733);
final Builder builder23 = new Builder("com.fasterxml.jackson.core");
builder23.open(true);
builder23.requires(new ModuleDescriptor.Requires[] { Builder.newRequires(of12, "java.base") });
Files
diffoscope-all.class.txt diffoscope-default.class.txt
From the Linux tar.gz release for 1.5.0-beta1, this .zip contains SystemModules$default.class and SystemModules$all.class extracted from the modules file using jimage extract and located at java.base/jdk/internal/module:
From my build using the guide docs/reprocudible.md, this .zip contains SystemModules$default.class and SystemModules$all.class extracted from the modules file using jimage extract and located at java.base/jdk/internal/module:
Ran into same issue on Debian GNU/Linux 11. Kernel 5.10.0-8-amd64. See #261 for details.
This issue has now been fixed upstream in https://bugs.openjdk.java.net/browse/JDK-8275509
Will unfortunately need to wait for the release of JDK 18 in April 2022 to incorporate this fix into Sparrow, but at least a solution is on the way.
I always thought of the error as an Debian related issue as @nyxnor and I experienced it on it?! For me the built went well on Linux Mint but failed (i.e. with some diffs) on the same physical machine once I installed Debian. On both machines I followed the same guides for building. But in the above mentioned issue a reproducible built seems possible on Debian [edit: acc. to a clarification the mentioned build wasn't done on a Debian but Ubuntu]...so not sure if the difference between the distros is important?!
Lets hope the JDK18 resolves the issue for good.
As the problem persists for me (even under a new Kubuntu VM) I tried to update to a JDK 18. The java link you provided in the README informs that
24th July 2021: AdoptOpenJDK is moving to the Eclipse Foundation and rebranding. Our July 2021 and future releases will come from Adoptium.net
It's now rebranded as Temurin. Now, if I download the provided JDK18 on said page (Linux, x64) and switch to the new java version I get
$ java -version
openjdk version "18.0.1" 2022-04-19
OpenJDK Runtime Environment Temurin-18.0.1+10 (build 10.0.1+10)
OpenJDK 64-Bit Server VM Temurin-18.0.1+10(build 19.0.1+10, mixed mode, sharing)
Trying to compile I get the following errors though:
$ ./gradlew jpackage -info
Initialized native services in: /home/requestprivacy/.gradle/native
Found daemon DaemonInfo{pid=9043, address=[299e3c73-8806-4f26-a99e-ae4a3f43891e port:39997, addresses:[/127.0.0.1]], state=Idle, lastBusy=1651506870190, context=DefaultDaemonContext[uid=c3e1852d-66c2-47de-b631-1449d4da7e68,javaHome=/home/requestprivacy/Code/JavaJDKs/jdk-16.0.1+9,daemonRegistryDir=/home/requestprivacy/.gradle/daemon,pid=9043,idleTimeout=10800000,priority=NORMAL,daemonOpts=--add-opens,java.base/java.util=ALL-UNNAMED,--add-opens,java.base/java.lang=ALL-UNNAMED,--add-opens,java.base/java.lang.invoke=ALL-UNNAMED,--add-opens,java.base/java.util=ALL-UNNAMED,--add-opens,java.prefs/java.util.prefs=ALL-UNNAMED,--add-opens,java.prefs/java.util.prefs=ALL-UNNAMED,--add-opens,java.base/java.nio.charset=ALL-UNNAMED,--add-opens,java.base/java.net=ALL-UNNAMED,--add-opens,java.base/java.util.concurrent.atomic=ALL-UNNAMED,-XX:MaxMetaspaceSize=256m,-XX:+HeapDumpOnOutOfMemoryError,-Xms256m,-Xmx512m,-Dfile.encoding=UTF-8,-Duser.country=US,-Duser.language=en,-Duser.variant]} however its context does not match the desired criteria.
Java home is different.
Wanted: DefaultDaemonContext[uid=null,javaHome=/home/requestprivacy/Code/JavaJDKs/jdk-18.0.1+10,daemonRegistryDir=/home/requestprivacy/.gradle/daemon,pid=15127,idleTimeout=null,priority=NORMAL,daemonOpts=--add-opens,java.base/java.util=ALL-UNNAMED,--add-opens,java.base/java.lang=ALL-UNNAMED,--add-opens,java.base/java.lang.invoke=ALL-UNNAMED,--add-opens,java.base/java.util=ALL-UNNAMED,--add-opens,java.prefs/java.util.prefs=ALL-UNNAMED,--add-opens,java.prefs/java.util.prefs=ALL-UNNAMED,--add-opens,java.base/java.nio.charset=ALL-UNNAMED,--add-opens,java.base/java.net=ALL-UNNAMED,--add-opens,java.base/java.util.concurrent.atomic=ALL-UNNAMED,-XX:MaxMetaspaceSize=256m,-XX:+HeapDumpOnOutOfMemoryError,-Xms256m,-Xmx512m,-Dfile.encoding=UTF-8,-Duser.country=US,-Duser.language=en,-Duser.variant]
Actual: DefaultDaemonContext[uid=c3e1852d-66c2-47de-b631-1449d4da7e68,javaHome=/home/requestprivacy/Code/JavaJDKs/jdk-16.0.1+9,daemonRegistryDir=/home/requestprivacy/.gradle/daemon,pid=9043,idleTimeout=10800000,priority=NORMAL,daemonOpts=--add-opens,java.base/java.util=ALL-UNNAMED,--add-opens,java.base/java.lang=ALL-UNNAMED,--add-opens,java.base/java.lang.invoke=ALL-UNNAMED,--add-opens,java.base/java.util=ALL-UNNAMED,--add-opens,java.prefs/java.util.prefs=ALL-UNNAMED,--add-opens,java.prefs/java.util.prefs=ALL-UNNAMED,--add-opens,java.base/java.nio.charset=ALL-UNNAMED,--add-opens,java.base/java.net=ALL-UNNAMED,--add-opens,java.base/java.util.concurrent.atomic=ALL-UNNAMED,-XX:MaxMetaspaceSize=256m,-XX:+HeapDumpOnOutOfMemoryError,-Xms256m,-Xmx512m,-Dfile.encoding=UTF-8,-Duser.country=US,-Duser.language=en,-Duser.variant]
Looking for a different daemon...
The client will now receive all logging from the daemon (pid: 14141). The daemon log file: /home/requestprivacy/.gradle/daemon/7.1/daemon-14141.out.log
Starting 4th build in daemon [uptime: 9 mins 19.572 secs, performance: 98%, non-heap usage: 21% of 256 MiB]
Using 3 worker leases.
Now considering [/home/requestprivacy/Bitcoin/sparrow] as hierarchies to watch
Watching the file system is enabled if available
Starting Build
Settings evaluated using settings file '/home/requestprivacy/Bitcoin/sparrow/settings.gradle'.
Projects loaded. Root project using build file '/home/requestprivacy/Bitcoin/sparrow/build.gradle'.
Included projects: [root project 'sparrow', project ':drongo']
Now considering [/home/requestprivacy/Bitcoin/sparrow] as hierarchies to watch
> Configure project :buildSrc
Evaluating project ':buildSrc' using build file '/home/requestprivacy/Bitcoin/sparrow/buildSrc/build.gradle'.
Selected primary task 'build' from project :
:buildSrc:compileJava (Thread[Execution worker for ':buildSrc',5,main]) started.
> Task :buildSrc:compileJava UP-TO-DATE
Caching disabled for task ':buildSrc:compileJava' because:
Build cache is disabled
Skipping task ':buildSrc:compileJava' as it is up-to-date.
:buildSrc:compileJava (Thread[Execution worker for ':buildSrc',5,main]) completed. Took 0.04 secs.
:buildSrc:compileGroovy (Thread[Daemon worker Thread 3,5,main]) started.
> Task :buildSrc:compileGroovy NO-SOURCE
file or directory '/home/requestprivacy/Bitcoin/sparrow/buildSrc/src/main/groovy', not found
Skipping task ':buildSrc:compileGroovy' as it has no source files and no previous output files.
:buildSrc:compileGroovy (Thread[Daemon worker Thread 3,5,main]) completed. Took 0.001 secs.
:buildSrc:pluginDescriptors (Thread[Daemon worker Thread 3,5,main]) started.
> Task :buildSrc:pluginDescriptors UP-TO-DATE
Caching disabled for task ':buildSrc:pluginDescriptors' because:
Build cache is disabled
Skipping task ':buildSrc:pluginDescriptors' as it is up-to-date.
:buildSrc:pluginDescriptors (Thread[Daemon worker Thread 3,5,main]) completed. Took 0.001 secs.
:buildSrc:processResources (Thread[Daemon worker Thread 3,5,main]) started.
> Task :buildSrc:processResources UP-TO-DATE
file or directory '/home/requestprivacy/Bitcoin/sparrow/buildSrc/src/main/resources', not found
Caching disabled for task ':buildSrc:processResources' because:
Build cache is disabled
Skipping task ':buildSrc:processResources' as it is up-to-date.
:buildSrc:processResources (Thread[Daemon worker Thread 3,5,main]) completed. Took 0.003 secs.
:buildSrc:classes (Thread[Daemon worker Thread 3,5,main]) started.
> Task :buildSrc:classes UP-TO-DATE
Skipping task ':buildSrc:classes' as it has no actions.
:buildSrc:classes (Thread[Daemon worker Thread 3,5,main]) completed. Took 0.0 secs.
:buildSrc:jar (Thread[Execution worker for ':buildSrc' Thread 2,5,main]) started.
> Task :buildSrc:jar UP-TO-DATE
Caching disabled for task ':buildSrc:jar' because:
Build cache is disabled
Skipping task ':buildSrc:jar' as it is up-to-date.
:buildSrc:jar (Thread[Execution worker for ':buildSrc' Thread 2,5,main]) completed. Took 0.004 secs.
:buildSrc:assemble (Thread[Execution worker for ':buildSrc' Thread 2,5,main]) started.
> Task :buildSrc:assemble UP-TO-DATE
Skipping task ':buildSrc:assemble' as it has no actions.
:buildSrc:assemble (Thread[Execution worker for ':buildSrc' Thread 2,5,main]) completed. Took 0.0 secs.
:buildSrc:pluginUnderTestMetadata (Thread[Execution worker for ':buildSrc' Thread 2,5,main]) started.
> Task :buildSrc:pluginUnderTestMetadata UP-TO-DATE
Caching disabled for task ':buildSrc:pluginUnderTestMetadata' because:
Build cache is disabled
Skipping task ':buildSrc:pluginUnderTestMetadata' as it is up-to-date.
:buildSrc:pluginUnderTestMetadata (Thread[Execution worker for ':buildSrc' Thread 2,5,main]) completed. Took 0.007 secs.
:buildSrc:compileTestJava (Thread[Execution worker for ':buildSrc' Thread 2,5,main]) started.
> Task :buildSrc:compileTestJava NO-SOURCE
file or directory '/home/requestprivacy/Bitcoin/sparrow/buildSrc/src/test/java', not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no previous output files.
:buildSrc:compileTestJava (Thread[Execution worker for ':buildSrc' Thread 2,5,main]) completed. Took 0.0 secs.
:buildSrc:compileTestGroovy (Thread[Execution worker for ':buildSrc',5,main]) started.
> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory '/home/requestprivacy/Bitcoin/sparrow/buildSrc/src/test/groovy', not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no previous output files.
:buildSrc:compileTestGroovy (Thread[Execution worker for ':buildSrc',5,main]) completed. Took 0.001 secs.
:buildSrc:processTestResources (Thread[Execution worker for ':buildSrc',5,main]) started.
> Task :buildSrc:processTestResources NO-SOURCE
file or directory '/home/requestprivacy/Bitcoin/sparrow/buildSrc/src/test/resources', not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no previous output files.
:buildSrc:processTestResources (Thread[Execution worker for ':buildSrc',5,main]) completed. Took 0.0 secs.
:buildSrc:testClasses (Thread[Daemon worker Thread 3,5,main]) started.
> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.
:buildSrc:testClasses (Thread[Daemon worker Thread 3,5,main]) completed. Took 0.0 secs.
:buildSrc:test (Thread[Daemon worker Thread 3,5,main]) started.
> Task :buildSrc:test NO-SOURCE
file or directory '/home/requestprivacy/Bitcoin/sparrow/buildSrc/build/classes/java/test', not found
file or directory '/home/requestprivacy/Bitcoin/sparrow/buildSrc/build/classes/groovy/test', not found
Skipping task ':buildSrc:test' as it has no source files and no previous output files.
:buildSrc:test (Thread[Daemon worker Thread 3,5,main]) completed. Took 0.001 secs.
:buildSrc:validatePlugins (Thread[Daemon worker Thread 3,5,main]) started.
> Task :buildSrc:validatePlugins FAILED
Caching disabled for task ':buildSrc:validatePlugins' because:
Build cache is disabled
Task ':buildSrc:validatePlugins' is not up-to-date because:
Input property 'classes' file /home/requestprivacy/Bitcoin/sparrow/buildSrc/build/classes/java/main/org/gradle/sample/transform/javamodules/ExtraModuleInfoPlugin.class has changed.
Input property 'classes' file /home/requestprivacy/Bitcoin/sparrow/buildSrc/build/classes/java/main/org/gradle/sample/transform/javamodules/ExtraModuleInfoPluginExtension.class has changed.
Input property 'classes' file /home/requestprivacy/Bitcoin/sparrow/buildSrc/build/classes/java/main/org/gradle/sample/transform/javamodules/ExtraModuleInfoTransform$Parameter.class has changed.
Watching 29 directories to track changes
Watching 30 directories to track changes
:buildSrc:validatePlugins (Thread[Daemon worker Thread 3,5,main]) completed. Took 0.013 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':buildSrc:validatePlugins'.
> There was a failure while executing work items
> A failure occurred while executing org.gradle.plugin.devel.tasks.internal.ValidateAction
> Unsupported class file major version 62
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 1s
6 actionable tasks: 1 executed, 5 up-to-date
Watching 30 directories to track changes
Is this an expected behavior since Sparrows codebase needs to be configured to use JDK18? Or what do I have to consider for building Sparrow with the latest JDK?
Is this an expected behavior since Sparrows codebase needs to be configured to use JDK18?
I suspect ./gradlew clean jpackage would resolve this particular issue. But I haven't started testing on JDK18 yet - will do so soon.
I tried to upgrade to Temurin 18.0.1, and ran into the same issue. Unfortunately, the build tool Gradle does not yet support Java 18: https://docs.gradle.org/current/userguide/compatibility.html. Gradle 7.5 will support Java 18, and should be released imminently.
Sparrow v1.6.6 has been built with Temurin 18.0.1+10, and the docs have been updated accordingly. Hopefully this issue is now resolved, but I'll keep it open until there is more confirmation.
Can you assure it is reproducible for me to try to build again or is it an expectation?
Give it a shot, I think craig would be happy to get another feedback.
At least for me all diffs seem to be gone now. Tested with an intra-release craig provided here, since with release 1.6.6 there where still diffs - though according to craig not attributable to the java issue here.
user@host:~$ sudo apt-get install temurin-17-jdk=18.0.1+10
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Version '18.0.1+10' for 'temurin-17-jdk' was not found
https://github.com/sparrowwallet/sparrow/blame/master/docs/reproducible.md#L30