Distribute flashable zip for Android 7+ for broken UnifiedNlp integration

[xenithorb]

So I've recently migrated to Android 7.1.1 as LineageOS has picked up the ball and started producing builds for bacon again.

My update workflow has changed in the following respects:

1. Where I would have normally had Xposed+FakeGapps installed, I now use tingle, as it's the only reliable way I could currently find to allow spoof'd signatures - given that Xposed for 7+ doesn't exist yet. This numeral is merely informative.

2. The point of the issue: As you know, given the UnifiedNlp README and the patch you produced here you are at least aware of the issue and the simple fact that going forward, UnifiedNlp and therefore many apps will not work with the traditional "Just install the app" workflow, due to the fact that only privileged system apps may be a location provider in 7+.

Your solutions are:

1. Apply the patch (which you can only do if you intend to build your own ROM from scratch)

   • CyanogenMod/LineageOS will not be including this patch: https://review.cyanogenmod.org/#/c/177082/
   • Patching a ROM and building said rom is a non-trivial process for average users

2. As root, install GmsCore in /system/priv-app/ (You said it slightly different for UnifiedNlp-only but the problem remains the same, and I verified this works myself)

This leaves us with option #2, which if you're a nightly user (hint: No other user of LineageOS atm) is frankly really a pain. And worse, a conundrum:

1. microG / GmsCore still needs to be installed prior to all GCM apps, and if it's reinstalled could it lose those subscriptions?
2. Apps in /system/priv-app/ will be erased on upgrade

Given that, I think it would be really nice if you could produce a flashable .zip that also includes an addon.d script to "rollover" microG to the next upgrade like Gapps does it. (hopefully preserving the push subscriptions too) - note when I say "push subscription" I mean the apps listed under the "Google Cloud Messaging" menu and how that operates with the necessity of GmsCore having to be present first.

Bonus points: Perhaps a second zip could be produced that does what tingle is doing and one-shot both the patching of the framework to support spoof'd signatures, and also install microG in the correct area.

Thanks for all of your hard work! I've been using this for a long time now and it's otherwise has worked great and has given me great peace of mind.

[ale5000-git]

@xenithorb: I have already created it: microG unofficial installer

[xenithorb]

While I appreciate your work, I do find it a bit opinionated given the circumstances ( I really don't want google software installed, just microG ). I also don't think it removes the burden of this project to do something official that supports current and future versions of Android.

I think we need a zip that does, at minimum:

1. Installs microG components (only) GmsCore and GSF-proxy into /system/priv-app/
2. Persists those packages into an upgraded /system via addon.d method or other means.

[ale5000-git]

1. The problem is that some components haven't real open source alternatives that work flawlessly, so I opt for an hybrid way to "open source" as much as it can be now. The open source alternative for accessing the Play Store is not updated (excluding maintenance fixes). The zip created by me can be used also from mostly "noob" people (it is really easy to use), doesn't require any knowledge to be used, and just require less than 1 minute to be installed, it also remove GApps automatically. The most important thing is that it allow an hassle-free transition, allow to switch from GApps to microG almost without notice any difference (I mean no feature lost, the are many positive differences).

2. Done.

[xenithorb]

Yalp store on fdroid works great. Its a good alternative for seeking out updates and new software.

[ale5000-git]

@xenithorb: The problem is that there is the risk of getting your Google account banned, and if you want to use both it and Apps that need Google account you need to insert the account twice or use two accounts (both options are annoying and unfriendly), correct me if I'm wrong.

[xsmile]

I would like to provide another alternative using haystack as a patching toolkit and a patch I have recently implemented to use UnifiedNlp as a user app: https://github.com/Lanchon/haystack/issues/3#issuecomment-275887254

[xenithorb]

Well, here I am 1 nightly LineageOS update later, and I accidentally tripped up and did the exact thing I foresaw and was trying to prevent here. :(

I forgot to add GmsCore.apk back to /system/priv-app and as a result, after rebooting, it lost all the GCM subscriptions. Now I have to go and delete all of the GCM apps and reinstall every single one. As it turns out, backing them up and restoring doesn't work either so this will incur some amount of data loss... No good!

Anyone have a workaround for that? One of the other problems is I have no idea which apps actually can utilize GCM, so I have to go from memory of what might use it.

[xsmile]

@xenithorb:

Use fr.simon.marquis.preferencesmanager to manually remove GMS settings from apps you except to have push notifications. Follow this example:

• select an app, e.g. WhatApp
• select file com.google.android.gms.appid.xml
• remove all entries (appVersion, lastToken, ...)
• kill the app process

The next time the app is started, it will aquire a new GMS token and it will be listed in microG. The entries might differ for various apps, but you get the idea.

[xenithorb]

Ok, that worked for the most part. Not every app seems to follow that convention, but for those that didn't I was able to look a little deeper and usually find some gcm_register_id or some other attribute that I could delete. Most did trigger a re-registering when such settings was missing. Thanks! @xsmile

[LuccoJ]

I echo @xenithorb in finding such a zip desirable for myself, but strictly without any proprietary components included. If Yalp Store runs the theoretical risk of getting your Google account banned, then so will any other free Play Store replacement potentially developed in the future (including µg-based ones), because they'd potentially breach the very same ToS. This would mean that under @ale5000-git the zip would never possibly become fully free, even the day everything had a free alternative, and for my preferences, that is not a desirable route.

Besides, it's always possible to get Google Play apps from places like APKPure, which are just websites. There are potential security concerns but I think that specific one, for instance, has a good track record, and it is always possible in line of principle to verify signatures before installing.

Of course, that's me, and other people have different opinions, and if I wanted to I could learn to create my own zip, so I'm just putting my opinion out there, not trying to force change :-)

[xenithorb]

The only viable way to install this going forward on Nougat+ is with a zip, or by manually pushing the file to /system/priv-app (allwhile not forgetting to do that ever or your app data disappears!). That's why I believe the way forward is for microG to supply a flashable zip. If at all possible, it should:

1. Modify the framework.jar (as Tingle does) (can python scripts be used in flashable zips?)

   • This is because Xposed does not support Nougat and won't for the foreseeable future, and the previous method involved FakeGapps+Xposed. I actually find "the tingle method" of patching framework.jar cleaner, and would wish for that to be done during the flashing process.

2. Copy the main GmsCore.apk to /system//priv-app

3. No store needs to be forced, proprietary binaries should not be distributed, and the user can download their choice of store whether Yalp or just F-Droid. (I prefer both, as there are simply certain things you need from the play store from time to time, as not all FLOSS projects publish on F-Droid)

[LuccoJ]

@xenithorb to nitpick,it's not projects themselves that publish on F-Droid, generally, but the F-Droid maintainers (mostly one of them)... so if an actually FLOSS project is not on F-Droid, it could be because there is a long backlog, in which case anyone can help by submitting working metadata for the project to F-Droid. If there are different issues, on the other hand, then that means it usually boils down to the project not really being pure FLOSS but rather some kind of OSS-with-strings-attached, even if it's just by depending on a nonfree compiling environment.

[Shadow53]

FWIW, I provide a couple of zip files as well for people to install microG on nougat. One installs the Play Store, another FakeStore and leaves the user open to choose F-Droid, Yalp Store, or otherwise. Maybe I'll make an aroma one in the future, who knows. The zips do not currently have addon.d support, but I intend to add it as well.

I'm linking to the download page on the off chance I end up changing the link to the file: https://shadow53.com/no-gapps/downloads/

(For the record, I do not intend to compete with @ale5000-git. I just prefer to use my own solutions when I can and had my own basic zips when ale5000's was released)

[ale5000-git]

@Shadow53: There isn't any problem :)

PS: I intend to add support for configuration but I haven't had time yet. Edit: Added.

[ale5000-git]

@xenithorb: The latest version of my zip have addon.d support. Currently it backup only the apk of microG Service Core but once I get some reports that it works and some free time I will extend to all components.

[Nanolx]

I also made my own solution: https://github.com/Nanolx/NanoMod

Unlike the others it's a Magisk module, so it does not get overridden by ROM update, except you wipe /data, but it kicks in Magisk as hard-dependency. The github repository also contains framework-patcher.sh a Bash script that makes using haystack much easier (works on GNU/Linux, BSD, Mac OSX).

There are three packages:

• full package (with lots of extra stuff, not as relevant here I guess, ~180 mb)
• microG only package (~14 mb)
• F-Droid only package (~6 mb)

I modified microG GmsCore and Play Store, so that (in-)app-purchases work (I followed mar-v-in's instructions in issue #309 for that purpose).

The module supports a setup file /sdcard/.nanomod-setup which controls what app store to use, where you can choose none, Play Store or Yalp Store (for the microG only package, the full package supports for more options). See the README in the github repository.

[xenithorb]

What's the upgrade path for microG for your solution? I take it because they're signed differently, F-Droid isn't going to upgrade it as normal.

I switched to using Haystack myself, and I wrote a bash script that uses adb to shuffle things around and generally do what I need after a TWRP flash. That was my evidentual temporary solution.

Nice that it has an easy to grok config file!!

[Nanolx]

If you manually update it, you would loose ability to do (in-)app-puchases, as Play Store and microG GmsCore need to have same signature for that to work.

But the path is /magisk/NanoMod/system/priv-app/GmsCore in case you don't care.

Though, once there's an update, I'll push it to my github repo and then you could just use mod.sh microg for the microG package (or mod.sh zip for the full package) to create a new installable zip.

[xenithorb]

That seems a bit opinionated for my usage unfortunately. (Even though I'm testing it out right now)

1. My personal use case is F-droid + microG (so I am using separate microG and F-droid) modules that I generated from your repo.
2. I don't care about microG being signed the same as play, because I don't, nor will I ever use play.

I was hoping it wasn't a modified apk so that F-droid could update microG, as I have the microg.org F-droid repo installed and I use that to update it regularly to test builds. Unfortunately, for it to work (and the reason for this ticket) is that I have to move it over to /system/priv-app each time in order for it to work after updates now. Otherwise F-Droid upgrades it just fine. Otherwise everything else is nice - and it's a bit of a segue for me into Magisk, which I haven't done anything with before. Pretty neat if you ask me.

Side node: the framework-patcher.sh (haystack) doesn't seem to work after installing Magisk-12.0, it freaks out and quits on some java error

[Nanolx]

You do patch the files from TWRP and use the correct api level? Cause Magisk does nothing on /system, so it can't actually make a difference.

[xenithorb]

Figured out the problem, it was my own past corruption of java. False alarm.

Everything works ok. A few things aren't clear:

1. Does Magisk neeed to be flashed after every update? (I update lineage every week)
2. Does framework-patcher.sh still need to be done after each update? (Is there any way to avoid this?)

[Nanolx]

1. yes, as it modifies the boot image
2. yes, after every update, there's no practicable other way, except using crDroid or OmniROM, which com pre-patched with fake sign

[xenithorb]

So the only thing I really did was remove the necessity to do cp /sdcard/GmsCore.apk /system/priv-app/ in my script :/

I do quite like Magisk though, and your framework. Kudos!

We still need a solution to patch the *.jar files on the phone. There's a discussion about the possibility of doing that here: https://github.com/Lanchon/haystack/issues/5

[xenithorb]

You see the problem that I have with the manual nature of some of these items is that it's reduced microG to only being used by techies, and then only ones willing to plug their phones into their computer to do a convoluted update. Even non-techies that have techie friends won't be able to use this because - while the usual process was to get someone "in the know" to set you up for the first time - that's no longer viable because it basically breaks normal updater updating. (i.e. a normal user who just updates from within LineageOS' updater) - if they don't remember or know to re-flash Magisk or patch the framework.jar to accept spoofing, then microG stops working for them and they will probably not know why. Ultimately I want this to eventually pass the "wife test," so I can setup my wife's phone and have her update it as normally and it not break. A sad state of affairs we're in, IMHO.

[ale5000-git]

@xenithorb: I have a lot of ideas to simplify all things ( and pass the "wife test" ;) ), I just lack time currently. At the end patching will be just 3 click on the phone.

[Nanolx]

I created a on-device framework-patcher. See https://github.com/Lanchon/haystack/issues/5 for more information. This is the first try, so may not work for everyone. Please test and report back.

[ale5000-git]

For those that didn't noticed Tingle does already work when executed inside Android (it is still a bit slow though).

[Torvin]

@xenithorb I would really appreciate if you share some more info about the second option you mentioned in the issue description. I have microG installed with Tingle, but UnifiedNlp (coarse location) won't work ("The system did not bind the UnifiedNlp service"). Do I need to uninstall it first? Do I then just copy the apk to /system/priv-app/ after and reboot?

[ale5000-git]

@Torvin: Not "copy to" but "move to".

[Torvin]

@ale5000-git move? from where?..