divVerent
Remove some non-determinism
Your app is not built reproducible in F-Droid, but we continuously test older versions on https://verification.f-droid.org and would like more and more apps to become repro.
Looking at this report: https://verification.f-droid.org/io.github.divverent.aaaaxy_106300004.apk.diffoscope.html#lib-arm---v-a-libgojni.so---readelf---wide---notes---
based on https://f-droid.org/docs/Reproducible_Builds/#golang maybe you can something to https://github.com/divVerent/aaaaxy/blob/v1.6.0/Makefile or https://github.com/divVerent/aaaaxy/blob/v1.6.0/AndroidStudioProjects/AAAAXY/app/build.gradle#L48 or https://github.com/divVerent/aaaaxy/blob/v1.6.0/AndroidStudioProjects/AAAAXY/app/build.gradle#L78 or etc
First thing I see is that the two versions are built with different versions of Go:
@333333*@go1.23.6 70899 @333333*@go1.23.7
I doubt I can make that go away.
Trying to follow the recommendations though - how can I get an updated report after my changes?
We can run multiple times locally and compare.
Can we find the exact go version in code so we can match yours?
[rpolzer@brlogenshfegle ~]$ go version
go version go1.23.6 linux/amd64
Which just means I should go and update, so doing that now.
[rpolzer@brlogenshfegle ~]$ go version
go version go1.23.7 linux/amd64
I generally stick with the same minor version of Go until something requires me to go beyond, as they usually lose some platform support.
As for SDKs:
export ANDROID_HOME=$HOME/Android/Sdk
This is the following packages:
Path | Version | Description | Location
------- | ------- | ------- | -------
build-tools;30.0.2 | 30.0.2 | Android SDK Build-Tools 30.0.2 | build-tools/30.0.2
build-tools;30.0.3 | 30.0.3 | Android SDK Build-Tools 30.0.3 | build-tools/30.0.3
build-tools;32.0.0 | 32.0.0 | Android SDK Build-Tools 32 | build-tools/32.0.0
build-tools;32.1.0-rc1 | 32.1.0 rc1 | Android SDK Build-Tools 32.1-rc1 | build-tools/32.1.0-rc1
cmdline-tools;latest | 6.0 | Android SDK Command-line Tools (latest) | cmdline-tools/latest
emulator | 31.2.10 | Android Emulator | emulator
ndk;21.4.7075529 | 21.4.7075529 | NDK (Side by side) 21.4.7075529 | ndk/21.4.7075529
ndk;25.0.8775105 | 25.0.8775105 | NDK (Side by side) 25.0.8775105 | ndk/25.0.8775105
patcher;v4 | 1 | SDK Patch Applier v4 | patcher/v4
platform-tools | 33.0.1 | Android SDK Platform-Tools | platform-tools
platforms;android-21 | 2 | Android SDK Platform 21 | platforms/android-21
platforms;android-31 | 1 | Android SDK Platform 31 | platforms/android-31
platforms;android-32 | 1 | Android SDK Platform 32 | platforms/android-32
platforms;android-33 | 2 | Android SDK Platform 33 | platforms/android-33
platforms;android-34 | 3 | Android SDK Platform 34 | platforms/android-34
sources;android-21 | 1 | Sources for Android 21 | sources/android-21
sources;android-33 | 1 | Sources for Android 33 | sources/android-33
I have compileSdk 34 in the gradle file, so I assume that's what it's using; no idea which of the NDK version it's using.
I am open to changing those versions - which does F-Droid prefer?
Other than that, I don't see quite the point of reproducible builds - as I am too new and was forced into Play App Signing, I can't match the signing key that the Play Store distributes anyway, so there definitely can't be any cross-updating between the two systems (otherwise I'd actually love to distribute the Google Play APK on F-Droid, so each can freely update the other).
But even without that, I guess having some verification is nice to have, yes, and generating deterministic binaries also reduces update size.
For now testing that at least the Linux build has no randomness left:
$ for i in $(seq 1 5); do git clean -xdf && mak BUILDTYPE=release && cp aaaaxy /tmp/aaaaxy.$i; done
They have the same size but differ.
-00ef7800: 0800 8da2 735a b4b4 e286 e10e 0000 5e2c ....sZ........^,
+00ef7800: 0800 0ba3 735a b4b4 e286 e10e 0000 5e2c ....sZ........^,
Lots of similar differences where 8da2 becomes 0ba3.
The other thing is - this seems fragile. Don't verifiable builds break all the time when I update a build tool version but F-Droid does not, or vice versa?
At this point I wonder if it wouldn't be better to use F-Droid's own build system for my releases on github as well. Is there an easy way from a shell script to run a build under F-Droid's environment, even without building from fdroiddata (as then I first have to actually create the version stanza etc. first)?
Hm... I suspect this might be the embedded zip file with the assets.
This does store file dates, which git sets to current time when extracting.
Right now I am creating the zip file with Info-ZIP, which appears to have no way to not store file dates and times; is there a better tool for this? I can't find options to not do that in 7za's manpage either.
I found https://salsa.debian.org/reproducible-builds/strip-nondeterminism - which might fix the zip issue.
Can this tool be used in the F-Droid build environment, and if so, how?
Edit: with this tool, I got identical Linux binaries. I guess that's a good start.
I couldn't apply -buildvcs=false as ebitenmobile doesn't support it, but I do now have two APKs that are identical for all except the signing block (and if I extract the zip files, the contents are identical). That's a win, I guess. So I guess on next release we can try again?
The .aab files (the ones I upload to Google Play) also are 100% identical now on rebuilds.
Thanks, sounds good
I generally stick with the same minor version of Go until something requires me to go beyond, as they usually lose some platform support.
we don't have access to your local machine 😄 I meant grep it from somewhere, eg. https://github.com/divVerent/aaaaxy/blob/main/.github/workflows/go.yml#L21 (which one of the files has the right version?)
no idea which of the NDK version it's using.
is this kept up to date https://github.com/divVerent/aaaaxy/blob/main/.github/workflows/go-buildonly.yml#L51 ?
But even without that, I guess having some verification is nice to have
this, yes
Let's first see if the changes are sufficient.
As for the Go version - IMHO the proper place to look is the go.mod file, which lists the Go version in the line toolchain.
Sadly if a newer Go is installed, it'll use that instead, and that's what happened in this week's release - I am not going to make another release just for that, but in the future I can try keeping that line updated so you can use it.
Please do not use the github workflows files - they are rather the range of Go versions I want to test at.
In the future, version in go.mod will be authoritative for releases - I guess that means F-Droid can use this field too now then. But not yet with this week's release.
I'll do a new release on Monday - let's then see if the issue is resolved.
latest diff output looks a bit better: https://verification.f-droid.org/packages/io.github.divverent.aaaaxy/ 👍 but not yet perfect
Seems like there's only one section remaining. Let's see if I can figure out what that is.
Actually two. I doubt I can figure the .data.rel.ro one out, but this one seems more reasonable:
[cb0e25] R<A6>\n
H<FE>/3<DC>^W<EB><CC>P+<A1><91>3<9E><B6>G<93><B2>x<C1>L-<95>Rd_9<F6>[<D0>f<CF>^L^P^_<DB>t_T"<CC><F4>4<C2>^G <C2>J<81><CD>@"<CA>\n
K<80>^W<D9>TG<97>b<DA>5<C4><F4>D3<9F>e5#<89>e^Q9<A4>0<B1><90>e+\[Z+<94>dYJ+8<E7>z<AD><A7>L=^Rq<81>i<D3><DD><DA>\9L<B6>J<E3><B9>J81<E1><A4><E7><DA>q0\n
M^R<E9><90>U<DD>F<F5><F6>^F{<FF>IBE<F5><EA><F1><FF>d^AOggS
[cb0ec9] 7
[cb0ecf] 8<D1>VX^\^A
[cb0ed7] &<A9><E0><8F>^A>\Y\n
D^Q^\<B5><95>22<D2>^Te\n
o^Y^F<AB><D6> \<97>T<D3><F6><F2><FB><DB><CD>,<A2><9A><DE>e<9B>;^E<FD><BF>z<F5><EA><90>9<E6>E<98><FF><D5>A<98>^M
[cb0f27] h^B^C<CC><FD>[<B3><FC>
Note the OggS nearby. This is likely part of the embedded zip archive of the game, I presume? Let's see...
No... I can't seem to find a way to extract the data from this file. I'm gonna leave this issue open, but it's unlikely I can fix it.
Note that if it's due to the embedded zip archive, it maybe is due to a version difference in the advancecomp tool. The version I have is 2.5-1, not sure what's used on F-Droid.
Any chance you can upload the aaaaxy.dat file that's created in the F-Droid build? I'd like to compare that file to mine.
I just can't seem to be able to extract that from the .so, there must be some transformation happening to it that I do not know about yet.
Oh, the F-Droid build uses embed mode, not zip mode. This differs from the aaaaxy.apk I am distributing, as I distribute an all-platforms APK that has a separate aaaaxy.dat and four different .so files; on F-Droid OTOH I optimized for F-Droid by having four APKs, one per platform, and all the data linked into the binary without a zip wrapper so it's memory mapped and doesn't need to be decompressed.
So the binaries can never be identical.
And yet, your comparison shows them to be very close, which surprises me.
{"1746603840.9419317": {"diffoscope": {"Available-in-Debian-packages": ["coreboot-utils", "libxmlb-dev"], "External-Tools-Required": ["cbfstool", "lipo", "otool", "xb-tool"], "Missing-Python-Modules": ["r2pipe"], "VERSION": "240"}, "local": {"file": "unsigned/io.github.divverent.aaaaxy_106301944.apk", "packageName": "io.github.divverent.aaaaxy", "sha256": "8c85573d9016967d58d3d9d4aee305634d7feec8471e72ab9fe7ce5293a643a2", "timestamp": 1746603840.9419317, "versionCode": 106301944, "versionName": "1.6.194+20250423.3812.ce4a067b"}, "remote": {"file": "tmp/io.github.divverent.aaaaxy_106301944.apk", "packageName": "io.github.divverent.aaaaxy", "sha256": "2435120b48b5151c2f47f557993fd2c77647cf5d4116e63136d63240f9f9409f", "timestamp": 1746603847.9219842, "versionCode": 106301944, "versionName": "1.6.194+20250423.3812.ce4a067b"}, "result": "verification of APK with copied signature failed\nComparing reference APK to APK with copied signature...\nFailed to run diffoscope tmp/io.github.divverent.aaaaxy_106301944.apk", "url": "https://f-droid.org/repo/io.github.divverent.aaaaxy_106301944.apk", "verified": false}}
Can you provide me the APK you used for reference? Or was that also built using the F-Droid repository? It's not the aaaaxy.apk I have on Github, right?
If the use of the embed module is the problem, I can switch the F-Droid build to use a zip, however that'd slow down game startup.
So you say there are zero differences in the build environment between the two? Then I guess I can reproduce it if I build my own fdroidserver setup?
Still it'd help if I can have both APKs myself (and not just the F-Droid one). Can you provide that? I am considering filling the differing bytes with garbage to see what happens.
So you say there are zero differences in the build environment between the two?
Not quite, @eighthave can comment on that
Can you provide that?
Not that I know, no
The buildserver instances are not guaranteed to be bit-for-bit identical. So far we've concentrated on the things we know will affect the build. Part of this process is discovering what causes builds to be different, tracking all those, then figuring out how best to handle them.
I could try switching it to the zip file based build (as it's possible //go:embed is causing the difference, just based on where the bytes are in the binary).
I kinda doubt that, but it's possible. And sadly I can't reproduce this issue with native Linux binaries, and AFAIK I can't run the Android ones in a debugger to see what those bytes may be doing, so... kinda stuck there. Even with ghidra trying to decompile the binary, I am kinda stumped about those.
However, switching to that build makes delta updates larger (IIRC F-Droid doesn't use them anyway) makes the game load slower - so any way to try this out before actually committing it into the F-Droid repo? Like, can we make a branch and run your reproducibility tests on that before making it an actually released version?
Now all F-Droid builds seem to be equal. I still can't reproduce it fully on my computer though.
The disassembly is already identical except for addresses though - the most glaring difference is that F-Droid's build contains DefaultGODEBUG and mine does not.
Another issue is that go-licenses doesn't appear to work on F-Droid. This is less critical though as F-Droid links to the source code very prominently. Pipeline log: https://gitlab.com/divVerent/fdroiddata/-/jobs/10631280336
(Note the lack of messages like
W0710 06:37:39.786590 1865766 library.go:141] "github.com/pierrec/lz4/v4/internal/lz4block" contains non-Go code that can't be inspected for further dependencies:
that I get in my build from go-licenses - it's like go-licenses can't figure out module dependencies when run on F-Droid)