WIP: Simple reproducible builds

[ktecho]

This started using https://github.com/ZeusLN/zeus/pull/535 as base. The Dockerfile is a lot simpler because I'm using a reactnativecommunity/react-native-android image that has everything built in. Both .sh files are from the other PR and have several problems yet:

1- compile.sh and build.sh are almost the same, but build.sh has the IMAGETAG stuff removed because I wanted to make it work without it (see point 3).

2- This works just for Debug releases (assembleDebug), not Release (assembleRelease). This is because we need the same keystore used to build the releases. For Debug, it generates a new keystore, as we don't care if it's reproducible or not. But it builts the apk correctly.

3- It works just the first time. So if you don't have the container created yet, it creates it, uses it and it works. If it already exists, it may not work. For this, maybe the IMAGETAG stuff in compile.sh could be useful, but I don't think I made it work.

[kaloudis]

Wow, you're much further along than I thought! What we have to do now is remove the keysigning process from the assembleRelease process. That way users will generate the same exact binary minus the signature.

[ktecho]

Wow, you're much further along than I thought! What we have to do now is remove the keysigning process from the assembleRelease process. That way users will generate the same exact binary minus the signature.

But then people won't be able to compare the .apk from Playstore and get the same MD5, right? Not exactly sure how this works. It's been ages without me doing something Android related.

Edit: ok, not only Playstore, but the one you download from github or the webpage.

[kaloudis]

Wow, you're much further along than I thought! What we have to do now is remove the keysigning process from the assembleRelease process. That way users will generate the same exact binary minus the signature.

But then people won't be able to compare the .apk from Playstore and get the same MD5, right? Not exactly sure how this works. It's been ages without me doing something Android related.

Edit: ok, not only Playstore, but the one you download from github or the webpage.

There's tooling that separates the main package from the signed data on top of it. I believe apkdiff does this: https://github.com/daniellockyer/apkdiff

[ktecho]

I've simplified the process a lot so that it can also be used to compile a local version of the code (see README.md).

Right now it installs the node packages then it starts compile process, but it fails with:

2022-08-12T16:53:27.026+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] FAILURE: Build failed with an exception.
2022-08-12T16:53:27.026+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 
2022-08-12T16:53:27.026+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * What went wrong:
2022-08-12T16:53:27.026+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Execution failed for task ':app:bundleReleaseJsAndAssets'.
2022-08-12T16:53:27.027+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] > Process 'command 'node'' finished with non-zero exit value 1
2022-08-12T16:53:27.027+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 
2022-08-12T16:53:27.027+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * Try:
2022-08-12T16:53:27.027+0000 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Run with --stacktrace option to get the stack trace.  Run with --scan to get full insights.

Will try again later, but probably I'll not advance too much until I get unlimited data and a faster computer.

If anybody wants to try to see where the problem is, you can try putting this at the line 13 in build.sh:

./gradlew --stacktrace --scan assembleRelease && \

Another thing I want to try, is using the previous docker image for the React builder, as a few months ago I tried this and it was working (latest image is from 15 days ago). For that, you would need to change the first line in Dockerfile to:

FROM reactnativecommunity/react-native-android:5.4

[emanuelb]

Comparing built APK to APK from provider can be done with:

1. unzip both APK in different folders and run diff --brief --recursive ./LocalBuild ./FromProvider Then only files relates to signature should be shown.

2. running diffoscope https://diffoscope.org/ on both APK which will compare them and show the difference.

3. tool developed by f-droid used to compare APKs by omitting signature https://github.com/obfusk/apksigcopier

See related issue I opened a while ago with my RB attempts of Zeus wallet: https://github.com/ZeusLN/zeus/issues/898 and another issue of diffs between the APK from f-droid to APK on github (the APK on f-droid is built using their build server) https://github.com/ZeusLN/zeus/issues/463

[ktecho]

Thanks @emanuelb I'll have a look at it

[ktecho]

My mind is slow being on vacation... I forgot that I can test this in a remote server that I have with unlimited bandwidth...

It seems we're hitting this issue: https://github.com/crypto-browserify/pbkdf2/issues/111

SyntaxError: /olympus/zeus/node_modules/parse-asn1/node_modules/pbkdf2/lib/default-encoding.js: Unexpected token (5:36)
error SyntaxError: /olympus/zeus/node_modules/parse-asn1/node_modules/pbkdf2/lib/default-encoding.js: Unexpected token (5:36)

  3 | if (global.process && global.process.browser) {
  4 |   defaultEncoding = 'utf-8'
> 5 | } else if (global.process && global."v16.16.0") {

    |                                     ^

But here someone states that it's more likely that the culprit is rn-nodeify: https://github.com/crypto-browserify/pbkdf2/issues/97

@kaloudis I'm not sure why you were able to compile v0.6.5 without hitting this issue, but changing the version of the package from 10.2.0 to 10.3.0 as stated in the last link fixes the issue for this build environment.

[ktecho]

@kaloudis if you want me to create a PR to upgrade the version of the library, just tell me.

[kaloudis]

@ktecho you can include the library upgrade in this PR

[kaloudis]

Do you not have to remove the signing config block for assembleRelease to complete successfully?

https://github.com/ZeusLN/zeus/blob/master/android/app/build.gradle#L139

[ktecho]

Do you not have to remove the signing config block for assembleRelease to complete successfully?

https://github.com/ZeusLN/zeus/blob/master/android/app/build.gradle#L139

Yeah, probably you're right. I was trying to generate a self-signed certificate to be able to release, so the process is valid for the normal release and the apks only diff in the certificate.

I'm going to try and generate the release without the signing config block to finish this, and if it works fine, I'll let you or others accomodate the android part.

[ktecho]

It's working! :)

I've tested the build for v.0.6.5 and it works. The only changes I had to make is updating the library I was refering before to the newer version without the issue, and commenting this line in build.gradle so it doesn't try to sign the apks:

    buildTypes {
        debug {
            signingConfig signingConfigs.debug
            if (nativeArchitectures) {
                ndk {
                    abiFilters nativeArchitectures.split(',')
                }
            }
        }
        release {
            shrinkResources true
            minifyEnabled enableProguardInReleaseBuilds
            proguardFiles getDefaultProguardFile("proguard-android.txt"), "proguard-rules.pro"
            //signingConfig signingConfigs.release
        }
    }

@kaloudis I don't know what you want to do with this signing stuff, because I think you need signing the apk that goes to the Play Store. Can we have another type of assemble, so we have assembleDebug, assembleRelease, assembleReleaseNoSign?

Another one: is there an easy way to change the app-universal-release-unsigned.apk to something like zeus-v0.6.5-unsigned.apk? Maybe the build script should rename the apks to this naming?

So for the comparison of the apks: I'll take a look at that later on which are the differences, but right now there is a big difference in size for the apks:

-rw-rw-r-- 1 ktecho ktecho  46M jun 22 19:16 zeus-v0.6.5.apk
-rw-r--r-- 1 ktecho ktecho  43M ago 14 11:40 app-universal-release-unsigned.apk

The tools that @emanuelb told me we could use pulls some 400 to 800 MB of dependencies in my debian testing environment. I'll have a look later on which one is the lighter one so we recomment it to people verifying the builds. Just so we start thinking about it: maybe the apk that can be downloaded from the page should be generated using this procedure so the diff is exact?

[ktecho]

More findings: apkdiff cannot be used because it reports both apks are equal, and they're not. It has a list of extensions ignored, so for example, we have different .so library sizes, but apkdiff is not reporting this difference. There are other differences in .xml files that apkdiff also is not detecting. I'll give a look to the other tools recommended in this thread.

@kaloudis I'm getting big differences in the size of this libsifir library for v0.6.5. I think this is related to the react-native-tor. I don't know if the size difference is because it's built on different computers. If so, that could be a problem for this reproducible builds stuff...

ktecho@host:~/devel/apks/065built/lib/arm64-v8a$ ls -larth libsifir*
-rw-rw-rw- 1 ktecho ktecho 4,7M ene  1  1981 libsifir_android.so
ktecho@host:~/devel/apks/065oficial/lib/arm64-v8a$ ls -larth libsifir*
-rw-rw-rw- 1 ktecho ktecho 7,8M ene  1  1981 libsifir_android.so

And also:

ktecho@ktecho-XPS-13-9350:~/devel/apks/065built/lib/arm64-v8a$ strings libsifir_android.so | grep "clang version"
Android (7155654, based on r399163b1) clang version 11.0.5 (https://android.googlesource.com/toolchain/llvm-project 87f1315dfbea7c137aa2e6d362dbb457e388158d)
Android (6875598, based on r399163b) clang version 11.0.5 (https://android.googlesource.com/toolchain/llvm-project 87f1315dfbea7c137aa2e6d362dbb457e388158d)

ktecho@ktecho-XPS-13-9350:~/devel/apks/065oficial/lib/arm64-v8a$ strings libsifir_android.so | grep "clang version"
Android (7155654, based on r399163b1) clang version 11.0.5 (https://android.googlesource.com/toolchain/llvm-project 87f1315dfbea7c137aa2e6d362dbb457e388158d)
Android (6875598, based on r399163b) clang version 11.0.5 (https://android.googlesource.com/toolchain/llvm-project 87f1315dfbea7c137aa2e6d362dbb457e388158d)
Android (6875598, based on r399163b) clang version 11.0.5 (https://android.googlesource.com/toolchain/llvm-project 87f1315dfbea7c137aa2e6d362dbb457e388158d)

[kaloudis]

@kaloudis I don't know what you want to do with this signing stuff, because I think you need signing the apk that goes to the Play Store. Can we have another type of assemble, so we have assembleDebug, assembleRelease, assembleReleaseNoSign?

we could but we probably won't use the old assembleRelease process at all. What we're going to do is manually sign the binary after the fact.

[kaloudis]

More findings: apkdiff cannot be used because it reports both apks are equal, and they're not. It has a list of extensions ignored, so for example, we have different .so library sizes, but apkdiff is not reporting this difference. There are other differences in .xml files that apkdiff also is not detecting. I'll give a look to the other tools recommended in this thread.

@kaloudis I'm getting big differences in the size of this libsifir library for v0.6.5. I think this is related to the react-native-tor. I don't know if the size difference is because it's built on different computers. If so, that could be a problem for this reproducible builds stuff...

ktecho@host:~/devel/apks/065built/lib/arm64-v8a$ ls -larth libsifir*
-rw-rw-rw- 1 ktecho ktecho 4,7M ene  1  1981 libsifir_android.so
ktecho@host:~/devel/apks/065oficial/lib/arm64-v8a$ ls -larth libsifir*
-rw-rw-rw- 1 ktecho ktecho 7,8M ene  1  1981 libsifir_android.so

And also:

ktecho@ktecho-XPS-13-9350:~/devel/apks/065built/lib/arm64-v8a$ strings libsifir_android.so | grep "clang version"
Android (7155654, based on r399163b1) clang version 11.0.5 (https://android.googlesource.com/toolchain/llvm-project 87f1315dfbea7c137aa2e6d362dbb457e388158d)
Android (6875598, based on r399163b) clang version 11.0.5 (https://android.googlesource.com/toolchain/llvm-project 87f1315dfbea7c137aa2e6d362dbb457e388158d)

ktecho@ktecho-XPS-13-9350:~/devel/apks/065oficial/lib/arm64-v8a$ strings libsifir_android.so | grep "clang version"
Android (7155654, based on r399163b1) clang version 11.0.5 (https://android.googlesource.com/toolchain/llvm-project 87f1315dfbea7c137aa2e6d362dbb457e388158d)
Android (6875598, based on r399163b) clang version 11.0.5 (https://android.googlesource.com/toolchain/llvm-project 87f1315dfbea7c137aa2e6d362dbb457e388158d)
Android (6875598, based on r399163b) clang version 11.0.5 (https://android.googlesource.com/toolchain/llvm-project 87f1315dfbea7c137aa2e6d362dbb457e388158d)

doesn't seem like a big deal, as long as the Tor library is still working (decreasing the size of the binary is a good thing). we shouldn't be comparing the official binary anyway, let's start comparing the newly built images across different machines. Can you give us a hash of the newly built version on the v0.6.5 tag and I'll try to do the same on my machine?

[kaloudis]

please commit the build.gradle changes first. want to make sure we're building everything exactly the same

[ktecho]

please commit the build.gradle changes first. want to make sure we're building everything exactly the same

Done.

user@host:/home/user/devel/zeus# md5sum android/app/build/outputs/apk/release/app-universal-release-unsigned.apk 
91f02cb94b293bd3ef388fdb84654037  android/app/build/outputs/apk/release/app-universal-release-unsigned.apk

[ktecho]

@kaloudis I just tried installing the apk in my smartphone and it says that the package is not valid, so it seems that more tests will be needed...

[ktecho]

This is the list of files that are different between v0.6.5 oficial and the auto-built one:

I guess the .so are not a problem, as you said. XMLs are not viewable and diffable by hand, so it's difficult to compare them. I would point at index.android.bundle, but I don't know how to know what's the problem. It's weird that android-development-tools generate the apk and the smartphone says it's invalid.

[emanuelb]

diffoscope tool can be used to view the difference between APK (both .so files, and the android binary xml files when compared from APKs), currently it doesn't support hermes bytecode, thus details on how to diff index.android.bundle are in the related issue: https://salsa.debian.org/reproducible-builds/diffoscope/-/issues/248

the below issue contain info on how to compare binary android xml by hand/themself: https://salsa.debian.org/reproducible-builds/diffoscope/-/issues/118

[kaloudis]

@kaloudis I just tried installing the apk in my smartphone and it says that the package is not valid, so it seems that more tests will be needed...

I think it needs to be signed since it's not in debug mode. See https://stackoverflow.com/a/24833908 for signing after building

[ktecho]

@kaloudis I just tried installing the apk in my smartphone and it says that the package is not valid, so it seems that more tests will be needed...

I think it needs to be signed since it's not in debug mode. See https://stackoverflow.com/a/24833908 for signing after building

Weird. After signing using that way, my smartphone starts the installation process, then it says Cannot install app because of a conflict with a package (that's approximate because it's in spanish). I read that it could be because the certificate of the installed app doesn't match with the one I'm installing, so I uninstalled the oficial app and tried to install it again. Now a different message appears: App has not been installed because it seems the package is not valid.

If anybody wants to test this, this is how I did it:

keytool -genkey -v -keystore zeus.keystore -alias zeus -keyalg RSA -keysize 2048 -validity 10000
jarsigner -verbose -sigalg SHA1withRSA -digestalg SHA1 -keystore zeus.keystore app-universal-release-unsigned.apk zeus

[emanuelb]

It might be because APK signing V2+ is needed while jarsigner support only APK V1 signing, thus usage of apksigner tool is needed https://developer.android.com/studio/command-line/apksigner.html

[ktecho]

It might be because APK signing V2+ is needed while jarsigner support only APK V1 signing, thus usage of apksigner tool is needed https://developer.android.com/studio/command-line/apksigner.html

It seems you're right:

ktecho@host:~/devel/apks$ apksigner verify app-universal-release-unsigned.apk 
DOES NOT VERIFY
ERROR: Target SDK version 30 requires a minimum of signature scheme v2; the APK is not signed with this or a later signature scheme

I'll try to sign the apk with this apksigner and report back. Thanks!

[ktecho]

I has worked!! But with a few caveats:

1- You must first uninstall currently installed app, so you'll lose your settings. This is not good. 2- While installing, you'll see a message saying "Locked by Play Protect. Play Protect don't know the developer of this app, so it's probably not sure to install it". You can click a link "Install it anyways", and it will install. 3- After install, Play Protect will fire again and say "Do you want to send this app to be analyzed? Protect yourself and other users sending the app to do a security analysis". I clicked over "Don't send".

After that, the app is installed. I configured my nodes again and they work (I tested LND with Tor and clearnet).

I don't know if we can still improve this behaviour... I guess the main problem is that the certificate used to sign Zeus currently doesn't match with the one that's generated. Can we do something about this?

[kaloudis]

I has worked!! But with a few caveats:

1- You must first uninstall currently installed app, so you'll lose your settings. This is not good. 2- While installing, you'll see a message saying "Locked by Play Protect. Play Protect don't know the developer of this app, so it's probably not sure to install it". You can click a link "Install it anyways", and it will install. 3- After install, Play Protect will fire again and say "Do you want to send this app to be analyzed? Protect yourself and other users sending the app to do a security analysis". I clicked over "Don't send".

After that, the app is installed. I configured my nodes again and they work (I tested LND with Tor and clearnet).

I don't know if we can still improve this behaviour... I guess the main problem is that the certificate used to sign Zeus currently doesn't match with the one that's generated. Can we do something about this?

Not much we can do here. It's not a big deal though IMHO as we will continue to sign builds with the existing Zeus key.

[ktecho]

For the people in the Play Store, there are no changes. For the people that want to use the reproducible build, they'll have to uninstall the app and install the built apk and reconfigure connections. I think I'll put a note stating that you'll have to do this.

I was thinking in something: if each time someone wants to build a version, a new certificate is created, 2 things happen:

1- Diffs will never match 100% between the published one and the one you're doing yourself (maybe not a big problem, but still) 2- Maybe when Zeus releases next version, Android will screw you again making you uninstall the previous version before you install the next one, losing the connection details again, because certificates don't match.

Maybe we could generate a certificate for the Android reproducible builds and upload to the repository, so you're always signing with the same certificate. This solves point 1 and 2. Maybe this way when Zeus release v0.6.6, you can do it just by tapping the .apk...

I think we should test how this upgrade process works, and we still need to test this in different machines.

I'll finish the script and push it here later or tomorrow.

[kaloudis]

Maybe we could generate a certificate for the Android reproducible builds and upload to the repository, so you're always signing with the same certificate. This solves point 1 and 2. Maybe this way when Zeus release v0.6.6, you can do it just by tapping the .apk...

I don't think this is a good idea - it could potentially lead to scenarios where people are fooled into installing malicious builds signed by this key.

People who want to build and install their own reproducible builds are going to have to accept the following:

1. you're going to have to reconnect your nodes and set your settings after moving away from the official release
2. you're going to have to manage your own signing keys

Both points are mitigated if you just install the official version. This does not prevent you from reproducing and comparing the builds - you can do that and just install the official version afterwards.

[kaloudis]

RE: testing

the Intel machine I got my hands on doesn't have enough RAM. I will get a VPS/cloud instance to test on in the coming days. I'm optimistic that the hashes will match.

[ktecho]

RE: testing

I'm optimistic that the hashes will match.

I think they won't, because of the different certificate :)