LinwoodDev
[Feature request]: Add Butterfly to IzzyOnDroid
Is your feature request related to a problem? Please describe
No response
Describe your feature request!
Add Butterfly to Accrescent so that it can have the convenience of an app store without the 10 days of waiting for release after it is published.
Additional context
For example, 2.4.0 Beta 1 was released on July 14th, and appeared on F-Droid on July 24 (10 days later). This means that if a bug was fixed, users would still experience/complain about it even after being fixed.
Code of Conduct
- [x] I agree to follow this project's Code of Conduct
Hello, thanks for your idea. I'm always open for new stores if they don't add more maintenance. The f-droid delay was a problem time to time and I see the problem here.
I'm also thinking about izzyondroid https://apt.izzysoft.de/fdroid/, I think this could be also a good alternative. I haven't heard about Accrescent. On the first view, it looks like it isn't really an "f-droid repository" and needs maintenance.
Personally, I'm not against reuploading my app on other appstores. If you do this, I would encourage showing it as "unofficial".
you can follow it here: https://gitlab.com/IzzyOnDroid/repo/-/issues/827
izzy on droid won't accept us due to the binary size I can't sadly reduce. Need to see what else i can do
I've already pointed you to Strategies To Reduce Flutter App Binary Size. Another useful source is Reduce your app size. If you scroll down there, you see a section stating "Avoid extracting native libraries" – you already do that, your native libraries are stored uncompressed. Reverting that, would bring down the APK size by approx. 10 MB (by setting useLegacyPackaging to true in your build.gradle.kts). The only disadvantage of that would be the libraries would need to be extracted on-device when the app is installed/updated, thus needing extra space there – at the advantage of less data transfer.
Not saying you have to go that path, just mentioning that it does exist, and you have that option.
Thanks for pointing it out. I tried the strategies but it only gave ~1mb less. I analyzed the apk a bit and found out that most of the size comes from native libraries. 10MB would be fantastic I think. But useLegacyPackaging sounds like it's outdated and shouldn't be used? Should I make a new buildconfig for useLegacyPackaging?
We still see many apps using that. But of course feel free to make a separate build with that, just with the arm64 libs for now, and give it a dedicated name at releases so we can pick it. So folks can chose. Those using Obtainium might then opt for the bigger APK (if you place a proper hint), while those going via IzzyOnDroid would get the smaller APK which then needs to have the libs extracted on-device. One can always switch between the two, as both would be signed by you.
An update from my side: I added the build option to the build.gradle.kts and added these binaries to the github actions in ab84a977b26c2b17116ba7135968b9dfde4eb7e9. The next nightly will be released tomorrow with the new apks and the stable this month.
I tested it and got an apk with around 18 MB.
About the size I expected (I just named a more conservative number: always nicer to get a positive surprise, right? 😉 ) Great to read it worked out! We'll wait for the stable one then I'd suggest. Gives also a chance to your testers to see if they note any difference. Thanks a lot!
Thanks! We've succeeded building it, but we can't get it RB:
-rw-r--r-- 0.0 unx 32544 b- 15630 defN 1981-01-01 01:01:02 4b9f6800 lib/arm64-v8a/libimage_processing_util_jni.so
- -rw-r--r-- 0.0 unx 4424 b- 1643 defN 1981-01-01 01:01:02 5ccfa207 lib/arm64-v8a/libirondash_engine_context_native.so
- -rw-r--r-- 0.0 unx 1111200 b- 560446 defN 1981-01-01 01:01:02 bd62de3d lib/arm64-v8a/libsuper_native_extensions.so
+ -rw-r--r-- 0.0 unx 4496 b- 1639 defN 1981-01-01 01:01:02 82b7affc lib/arm64-v8a/libirondash_engine_context_native.so
+ -rw-r--r-- 0.0 unx 1108408 b- 559576 defN 1981-01-01 01:01:02 91f6460c lib/arm64-v8a/libsuper_native_extensions.so
-rw-r--r-- 0.0 unx 4896 b- 1756 defN 1981-01-01 01:01:02 567e9a21 lib/arm64-v8a/libsurface_util_jni.so
The difference in the native libs is strange. Looks like there's some Rust involved, but I found no such indication in your repo:
Same results whether we build on Debian bookworm, or Ubuntu noble (which your GH action says you are using). I was orienting at your Github action when setting up our build. Did I miss something? In the attached ZIP, you'll find the full diffs, and also the build recipe we've used. In case you wonder at the latter: yes, took me some fiddling around to get rid of your enforced signing 🙈 but we need to build an unsigned APK to confirm RB – not one signed with a debug key…
The thing is: I use super_clipboard which uses super_native_extensions that has rust under it. I will look at it tomorrow: https://github.com/superlistapp/super_native_extensions
Strange that your build pulls in a different version than ours. But yes, one of the two affected native libraries indeed is libsuper_native_extensions.so, so that matches. Thanks for looking into it!
I don't find a solution but I think the binaries gets built different on different ndk versions: https://github.com/superlistapp/super_native_extensions/issues/548#issuecomment-3342202400...
That's then basically the solution you point to: not using the pre-built binaries, but building them yourself. Would that be an option for you?
I'm not sure what you mean? You want to build the apk yourself or using the prebuilt .so files? But I'm not sure how it should work
We're using Reproducible Builds (see e.g. Reproducible Builds, special client support and more at IzzyOnDroid for some background) to confirm whether an app was really built from the source it claims to, thus increasing trust. So we need to come up with a byte-identical app. Ideally, everything is built from source then (and no pre-built binaries) – as only what was built from source on both ends can be fully confirmed (and you then also know the libs are fine, and there was no "supply chain injection" on them).
I didn't check how you handle those libs now, we just figured those are the only differences and you mentioned different builds due to different NDK versions. Just looking how to best align our builds, to end up with a confirmed RB.
As mentioned here, this is the build recipe I've used last:
build:
- sed -r 's/signingConfigs.getByName.*/null/ ; /^ signingConfigs \{/,/^ \}/d' -i app/android/app/build.gradle.kts
- FLUTTER_VERSION=$(sed -rn 's/^\s*flutter\:\s*([0-9.]+)$/\1/p' app/pubspec.yaml)
- cd app
- git clone -b $FLUTTER_VERSION https://github.com/flutter/flutter
- export PUB_CACHE=$(pwd)/.pub-cache
- flutter/bin/flutter config --no-analytics
- flutter/bin/flutter pub get
- USE_LEGACY_PACKAGING=true flutter/bin/flutter build apk --release --target-platform android-arm64 --split-per-abi --flavor production
(the first line is to get rid of the signing, as we need to build an unsigned APK)
It sounds good to me, it's the same command as in https://github.com/LinwoodDev/Butterfly/blob/9bbfe93a17422a362b8a3dd6be2230ccbc824cb0/.github/workflows/build.yml#L65.
Is the problem of super-natives still present?
I looked a bit and these libs are written in rust: github.com/superlistapp/super_native_extensions and https://github.com/irondash/irondash. I looked around and found out that rust isn't really reproducable and deterministic when building things: https://internals.rust-lang.org/t/reproducible-builds-for-rustc-gsoc-25-idea/22532, https://github.com/rust-lang/rust/labels/A-reproducibility.
I'm not sure if I can fix it myself but I'm open for ideas. Otherwise we need to exclude these libs?
What I also found are github artifact attestations to verify that a built is verified on the github ci: https://docs.github.com/en/actions/how-tos/secure-your-work/use-artifact-attestations. But I'm not sure if this could be used here?
that rust isn't really reproducable and deterministic when building things
Oh, that depends. It's more tricky than Kotlin, Java, or even Flutter – but we do have several apps with Rust components building reproducibly here.
github artifact attestations to verify that a built is verified on the github ci
Nothing we can use for that, sorry. Reproducible Builds mean we get a byte identical APK as result – not someone saying "I approve".
So back to the roots: where in the process are those libraries built? Neither our recipe, nor your workflow do mention building libraries in Rust. To get Rust code build reproducibly, we again need to match build paths and Rust versions. But if your build downloads pre-build libraries, our build should download the same – except the target is moving (something like "latest" won't work, because at the time we build, it might already be a newer version). I've searched your repo for "rust", but all that turned up in your code was "trust" 🤷♂️
Last release I've checked was 2.4.0 (as we pick only releases, not pre-releases). But I can of course try with the last pre-release, just to see if the problem remained.
APK appid mismatch: expected dev.linwood.butterfly, got dev.linwood.butterfly.nightly
Hm, OK… Well, looks like https://github.com/LinwoodDev/Butterfly/releases/download/v2.4.1-rc.3/linwood-butterfly-android-arm64-legacy.apk was not built from the commit the tag points to. This time, not only the native libs differ, but also the AndroidManifest.xml does. Err, wait: the nightlies have a different build recipe… oh my… Ah, crap, now of course the resulting APK also has a different name: different flavor, I should have seen that coming. OK, yet another round then…
OK, done. Only difference remaining are the native libraries again. As it seems to be mostly a sorting thing, it might have to do with the number of threads used during build. So let's run a last round, limiting our build to 4 CPU cores (the same Github does). Rolling dice, maybe, but we might be lucky there… Nope, no dice unfortunately 😢
Hmm, then i don't have any idea. The build files from super_native_extensions are:
- https://github.com/superlistapp/super_native_extensions/blob/main/super_native_extensions/android/build.gradle
- https://github.com/superlistapp/super_native_extensions/blob/main/super_native_extensions/cargokit/gradle/plugin.gradle
- https://github.com/superlistapp/super_native_extensions/blob/main/super_native_extensions/cargokit/build_tool/lib/src/build_gradle.dart
I still do not fully understand how these libraries get into your app. Are they part of your build process (and if so, how is the Rust version evaluated) – or do you pull in pre-compiled binaries (in which case I wonder why it would be different ones for our build)?
They are all inside the gradle build as subprojects that have their own building pipeline.
For irondash I have prebuilt binaries: https://github.com/CodeDoctorDE/irondash/releases/tag/precompiled_bf8aa1005316238284c3d456920c6e53. But I think the super_native_extension will be built by downloading rustup, installing rust and building the source code
And they probably do not specify a Rust version – at least I could not find that. Only the vague specifications "stable" and "nightly", which are both moving targets. No specific version locked. So if we build with different versions, that might explain differences. They describe here how to configure cargo for it, and seem to use precompiled binaries themselves. But there it only sets a prefix; which version they use, or where that is defined (i.e. how the final URL is built), I cannot find. Which makes RB very difficult, to put it softly.
In our recipe, we've already matched the build env as close to Github's as we can: homedir, build path, android home, even the OS image (Ubuntu noble). I don't find any hints to Rust in the build logs. And the only mention of those super_native_extension I could find in the logs are:
+ build_tool 1.0.0 from path /home/runner/work/Butterfly/Butterfly/app/.pub-cache/hosted/pub.dev/super_native_extensions-0.9.1/cargokit/gradle/../build_tool
Note: /home/runner/work/Butterfly/Butterfly/app/.pub-cache/hosted/pub.dev/super_native_extensions-0.9.1/android/src/main/java/com/superlist/super_native_extensions/DragDropHelper.java uses or overrides a deprecated API.
That probably leaves us both clueless now I guess?
Im also not really sure. I found this on github https://github.com/superlistapp/super_native_extensions/blob/main/super_native_extensions%2Frust%2Fcargokit.yaml.
If we find a way manually building the libs we could try to modify the pipeline... I think if we download rust previously we could specify the rust version... But otherwise im also clueless. I hope the new native asset feature in flutter will help here
I think if we download rust previously we could specify the rust version...
That's one thing I was hoping to find in the logs: where Rust was being installed (we'd need to match its path as well). Unfortunately, that part is not logged.
I found this on github
So did I (linked in my previous comment). Unfortunately, it's just a prefix – and I've no clue where the complete path is built and based on what. Looking at the base path, that could be supposed to match the tag name – probably fetching the latest tag matching that prefix, and using the pubkey + sig file to ensure the downloaded file is fine. But then, we both should end up with the exactly same files: their latest release is from June.
That said, the supernative libs are not the only ones affected. What differs are:
-rw-r--r-- 0.0 unx 2571292 b- 1178278 defN 1981-01-01 01:01:02 c47206ca classes.dex
- -rw-r--r-- 0.0 unx 16384944 b- 6562228 defN 1981-01-01 01:01:02 11e6aa67 lib/arm64-v8a/libapp.so
+ -rw-r--r-- 0.0 unx 16384944 b- 6562206 defN 1981-01-01 01:01:02 3c5e2233 lib/arm64-v8a/libapp.so
-rw-r--r-- 0.0 unx 7112 b- 2630 defN 1981-01-01 01:01:02 4f56ac38 lib/arm64-v8a/libdatastore_shared_counter.so
-rw-r--r-- 0.0 unx 11037032 b- 5173509 defN 1981-01-01 01:01:02 ef38a4ae lib/arm64-v8a/libflutter.so
-rw-r--r-- 0.0 unx 32544 b- 15630 defN 1981-01-01 01:01:02 4b9f6800 lib/arm64-v8a/libimage_processing_util_jni.so
- -rw-r--r-- 0.0 unx 4424 b- 1643 defN 1981-01-01 01:01:02 8ebf4534 lib/arm64-v8a/libirondash_engine_context_native.so
- -rw-r--r-- 0.0 unx 1109416 b- 559161 defN 1981-01-01 01:01:02 f68ce12b lib/arm64-v8a/libsuper_native_extensions.so
+ -rw-r--r-- 0.0 unx 4424 b- 1650 defN 1981-01-01 01:01:02 a2761fda lib/arm64-v8a/libirondash_engine_context_native.so
+ -rw-r--r-- 0.0 unx 1108408 b- 559576 defN 1981-01-01 01:01:02 91f6460c lib/arm64-v8a/libsuper_native_extensions.so
-rw-r--r-- 0.0 unx 4896 b- 1756 defN 1981-01-01 01:01:02 567e9a21 lib/arm64-v8a/libsurface_util_jni.so
Now, let's take a look at the diffoscope. libapp.so is yours. Fun: where your build states "Butterfly Nightly", ours just has "Butterfly" (plus the names show in different places). Next, libirondash seems to use a different clang version (18.0.1 in your build, 19.0.1 in ours; something updated within the week between our builds, and the version was not pinned?) For libsuper, I see a lot of ordering differences – but hey, look there: rustc·version·1.90.0 in your build, rustc·version·1.87.0 in ours. So the Rust version is obviously not pinned. But we still have no clue how it's evaluated: even if we pre-install Rust (to what path?), their build might decide to use a different one. Might be something to file in their issue tracker: pin versions so the libs can be built reproducibly?
Giving you the diff files here, so you can take a look for yourself: diff.zip