[FEATURE] Reproducible Builds

[IzzySoft]

Describe the feature

At IzzyOnDroid we support Reproducible Builds (see: Reproducible Builds, special client support and more at IzzyOnDroid). Trying for yours, I was able to successfully generate the APK (going by your Github action for orientation), but the resulting APKs were not identical. Differences were rather plenty:

apk.diff.gz

What did I miss? For one (but that's unlikely to be the only reason), I see that you use JDK 18 in your Github action. That's not an LTS. It's OK for text/debug builds, but I hope for release builds you use an LTS release (e.g. 17 or 21)?

We'd appreciate if you could help making your build reproducible. We've also prepared some hints on reproducible builds for that.

Looking forward to your reply!

Why would it be useful to add?

See the link above: RBs confirm the APKs shipped were indeed built from the source they claim – with nothing removed, added or altered. So they count as a security feature.

Example(s)

n/a

Additional context

Build recipe used here:

build:
  - sed -r '/signingConfig = releaseSigning/d' -i app/build.gradle.kts
  - sed 's/include\(.*\)/include("arm64-v8a")/' -i app/build.gradle.kts
  - chmod +x gradlew
  - ./gradlew assembleFossRelease -PsplitApks

(we remove the signing as we need an unsigned APK to compare against). Build here was done on the commit the release v7.1.0 points to.

[Mahmud0808]

I see that you use JDK 18 in your Github action. That's not an LTS. It's OK for text/debug builds, but I hope for release builds you use an LTS release (e.g. 17 or 21)?

Release builds are built locally on my pc and I use JDK 21 on Android Studio.

Edit: Can you kindly check the difference again? I have disabled crunchPngs, bumped JDK version to 21 and followed your build recipe. The difference in one unsigned APK built from Android Studio and another APK built from gradlew is only 4 bytes. But I can't get the full difference details as I am on a windows machine. Thank you.

[IzzySoft]

Release builds are built locally on my pc and I use JDK 21 on Android Studio.

👍 Will change the recipe to use 21 then.

Can you kindly check the difference again? I have disabled crunchPngs, bumped JDK version to 21 and followed your build recipe.

If you can provide me with an APK you've built from a clean tree at a commit including that (and let me know which commit it was), I can run it through the builder. But I can hardly compare against an APK I don't have 😉 You can rename the APK to .zip, then Github lets you attach it to a comment here.

he difference in one unsigned APK built from Android Studio and another APK built from gradlew is only 4 bytes. But I can't get the full difference details as I am on a windows machine.

Your question includes the answer: Windows. Probably 4 files in META-INF/services/ now (there were 2 before, which had \r\n with your build and \n with ours – i.e. Windows line-breaks versus Linux ones). Easy to fix on our end then.

[Mahmud0808]

If you can provide me with an APK you've built from a clean tree at a commit including that (and let me know which commit it was), I can run it through the builder. But I can hardly compare against an APK I don't have 😉 You can rename the APK to .zip, then Github lets you attach it to a comment here.

Here is the commit: 1d67288fc9c0c20c0093f4a3f363e93556b5a881 Here is the APK I built on my Android Studio (It's zipped, renaming didn't help): Iconify v7.1.0-foss-arm64-v8a-release-unsigned.zip

[IzzySoft]

It's zipped, renaming didn't help

Was too big, yeah. Note: Please avoid "blank characters" in file names (Iconify v7.1.0 => Iconify_v7.1.0), those are far too often problematic in scripting.

Still not RB, looks pretty much the same as before. Let me show you an excerpt:

  -rw-r--r--  0.0 unx   894160 b-   368472 defN 1981-01-01 01:01:02 d5b24482 lib/arm64-v8a/libzipalign.so
+ -rw-r--r--  0.0 unx     2261 b-      521 defN 1981-01-01 01:01:02 81ab8b18 assets/CompileOnDemand/android/PGB1/res/drawable/progress_horizontal_material.xml
+ -rw-r--r--  0.0 unx     2261 b-      521 defN 1981-01-01 01:01:02 81ab8b18 assets/CompileOnDemand/android/PGB1/res/drawable/progress_indeterminate_horizontal_material.xml
+ -rw-r--r--  0.0 unx     1102 b-      277 defN 1981-01-01 01:01:02 1276ba9c assets/CompileOnDemand/android/PGB1/res/drawable/seekbar_thumb_material_anim.xml
+ -rw-r--r--  0.0 unx     2088 b-      483 defN 1981-01-01 01:01:02 e925a7a9 assets/CompileOnDemand/android/PGB1/res/drawable/seekbar_track_material.xml
  -rw-r--r--  0.0 unx     2261 b-      521 defN 1981-01-01 01:01:02 81ab8b18 assets/CompileOnDemand/android/PGB10/res/drawable/progress_horizontal_material.xml
  -rw-r--r--  0.0 unx     2261 b-      521 defN 1981-01-01 01:01:02 81ab8b18 assets/CompileOnDemand/android/PGB10/res/drawable/progress_indeterminate_horizontal_material.xml
  -rw-r--r--  0.0 unx      949 b-      235 defN 1981-01-01 01:01:02 24d060f1 assets/CompileOnDemand/android/PGB10/res/drawable/seekbar_thumb_material_anim.xml
@@ -21,10 +25,6 @@
  -rw-r--r--  0.0 unx      949 b-      235 defN 1981-01-01 01:01:02 24d060f1 assets/CompileOnDemand/android/PGB11/res/drawable/seekbar_thumb_material_anim.xml
  -rw-r--r--  0.0 unx     1733 b-      464 defN 1981-01-01 01:01:02 32c208d3 assets/CompileOnDemand/android/PGB11/res/drawable/seekbar_track_material.xml
  -rw-r--r--  0.0 unx      862 b-      301 defN 1981-01-01 01:01:02 c2a28de2 assets/CompileOnDemand/android/PGB11/res/drawable/seekbar_track_material_iconify.xml
- -rw-r--r--  0.0 unx     2261 b-      521 defN 1981-01-01 01:01:02 81ab8b18 assets/CompileOnDemand/android/PGB1/res/drawable/progress_horizontal_material.xml
- -rw-r--r--  0.0 unx     2261 b-      521 defN 1981-01-01 01:01:02 81ab8b18 assets/CompileOnDemand/android/PGB1/res/drawable/progress_indeterminate_horizontal_material.xml
- -rw-r--r--  0.0 unx     1102 b-      277 defN 1981-01-01 01:01:02 1276ba9c assets/CompileOnDemand/android/PGB1/res/drawable/seekbar_thumb_material_anim.xml
- -rw-r--r--  0.0 unx     2088 b-      483 defN 1981-01-01 01:01:02 e925a7a9 assets/CompileOnDemand/android/PGB1/res/drawable/seekbar_track_material.xml
  -rw-r--r--  0.0 unx     2261 b-      521 defN 1981-01-01 01:01:02 81ab8b18 assets/CompileOnDemand/android/PGB2/res/drawable/progress_horizontal_material.xml

and then pick the interesting lines of an example:

+ -rw-r--r--  0.0 unx     2261 b-      521 defN 1981-01-01 01:01:02 81ab8b18 assets/CompileOnDemand/android/PGB1/res/drawable/progress_horizontal_material.xml
- -rw-r--r--  0.0 unx     2261 b-      521 defN 1981-01-01 01:01:02 81ab8b18 assets/CompileOnDemand/android/PGB1/res/drawable/progress_horizontal_material.xml

Now what does this show? Identical hash (81ab8b18), so these files are identical. Size, time, etc all match. But they appear in a different order in the APK. Any idea what might cause that?

built on my Android Studio

Hm… Which JDK? If you don't know, it's most likely Jetbrains (JBR) – which is known to cause issues with RB, though I cannot remember having seen this kind originating from it.

[Mahmud0808]

Now what does this show? Identical hash (81ab8b18), so these files are identical. Size, time, etc all match. But they appear in a different order in the APK. Any idea what might cause that?

Literally no idea, these are in assets directory of the APK. So they should be untouched.

Hm… Which JDK? If you don't know, it's most likely Jetbrains (JBR) – which is known to cause issues with RB, though I cannot remember having seen this kind originating from it.

[IzzySoft]

That's Jetbrains, as expected. And looking at ZIP ordering differences:

Bug: Android Studio builds have non-deterministic ZIP ordering

Solution (upstream): either build using the CLI (not Android Studio) or use Android Gradle plugin 7.1.X or later.

Building from CLI (i.e. at the command line, with gradle assembleRelease) was what I was going to recommend next. Which version of AGP have you with Studio? If it's older than 7.1, maybe update it – and switch to OpenJDK in Studio (can be done from within it) if building from CLI is not an option. As you have a Github action for debug builds: maybe use a similar one for release builds. You can build them unsigned there and manually sign them on your machine, so you don't need to trust Github with your credentials for that.

[Mahmud0808]

Which version of AGP have you with Studio?

It's 8.8.1, I always try to update it as soon as a new one is available.

switch to OpenJDK in Studio (can be done from within it)

I'll take a look.

if building from CLI is not an option. As you have a Github action for debug builds: maybe use a similar one for release builds. You can build them unsigned there and manually sign them on your machine, so you don't need to trust Github with your credentials for that.

I am actually trying to avoid manually singing each apk. And I want to handle release builds manually without using github actions. :(

[IzzySoft]

I am actually trying to avoid manually singing each apk. And I want to handle release builds manually

So manually without manually, OK, got it 🤪

You already have the signing information in your app/build.gradle.kts, so even running the build from CLI using gradle should handle that. With that properties file (containing the signing credentials etc) in place, that should be as simple as

chmod +x gradlew
./gradlew assembleFossRelease -PsplitApks

Ideally with using OpenJDK 21 then 😉 I could of course also offer you using our tools, which then even would make sure builds always happen in a clean tree at the commit you tag (they use ephemeral Podman (or if you prefer, Docker) containers for that). While their main purpose is to confirm RB, they can also be used to "just build that APK".

[IzzySoft]

v7.2.0 has the identical problem still.

[IzzySoft]

@Mahmud0808 any word on this? Do we still follow this, or shall we rather drop trying for RB?

[Mahmud0808]

I think we should drop trying for RB

[IzzySoft]

OK, I'll move it from the "waiting list" then. We can leave this issue open if you just want to skip this for now, but continue later (in which case you can drop me a note when I shall run another test) – or close it should you want to drop it altogether.

[Mahmud0808]

OK, I'll move it from the "waiting list" then. We can leave this issue open if you just want to skip this for now, but continue later (in which case you can drop me a note when I shall run another test) – or close it should you want to drop it altogether.

I will try in future, you can remove it from waiting list for now. Been very busy these days so don't get time for side projects unfortunately.

[IzzySoft]

Just done. Simply give me a ping when you want me to try again, and we can pick up again. Thanks!