Repository structure changes

[mar-v-in]

I was considering to do a major change in the structure of the microG repositories. The main reason therefor is that I often have to touch multiple repositories. Other reason is that the submodule/symlink system is causing problems every now and then (with non-unix systems, Android Studio, Kotlin gradle plugin, ...).

So I'd propose the following changes:

• [x] Merge GmsApi, GmsLib and GmsCore repos into a single new repository.
  • With each release form that repository (affecting all three components), upload as much as sensible and possible to maven central and GitHub releases.
• [x] Upload remote-droid-guard library (*), wearable library and unifiednlp-base to maven central and fetch from there during build of play-services-core
• [ ] Split play-services-core into smaller modules, that can also be included in third-party apps together with corresponding play-services library from microg, so they can work even on systems that don't have microG installed
  • This also implies more focus on play-services library (aka GmsLib) for better integration in third-party apps.
• [ ] Remove Android.mk build scripts from all repositories, as they tend to be incompatible with one or another android build version anyway.
  • Provide alternative repository with prebuilts from releases as well as other stuff required for proper
• [x] Rename repositories to get rid of android_packages_ prefix and alike as repositories are no longer intended to be included with android repo tool.

I think this will eventually make contributions from outsiders easier, but I am happy to hear other opinions on that from current contributors (@ale5000-git, @Nanolx, @Vavun, @voidstarstar, @chris42, @oldosfan, etc). Most of these changes will probably not happen before September given my current schedule.

(*) It's not functional for SafetyNet right now, but I will definitely have a look at it at some point again.

[chris42]

From my view, that are very good ideas. I fully support the streamlining of the repositories. I myself had a few headaches with the submodules, especially as some repositories are used multiple times e.g. GmsApi. Regarding the use of Maven, I don't know, as I have no experience in that (yet). As well as with the splitting of play-service-core. After building microg in my ROM, I agree with the removing of the Android.mk, renaming and moving to a prebuild setup ( discussed in parallel in #821 ) If you need support, let me know. I am probably not any help in the maven library stuff, but I am working on a prebuild repository and testing a few things in there.

[voidstarstar]

I do like the idea of separate apk modules/plugins for microG. This is especially useful for 3rd party dependencies that may be updated far more frequently than microG, that have critical security updates, or that are just large in size (e.g. see #802).

This is also beneficial when there are different implementations to choose from for a single service (e.g. maps). If there are multiple services, each with their own choice of implementation, we don't want to have an apk for each possible combination (e.g. GmsCore-v0.2.8.12345-mapbox-mozillanlp-someOtherServiceImpl-....apk), but due to reasons such as the size, we also don't want to include them all in a monolith apk.

It could be nice to have a centralized Activity in microG that will show a mapping from each supported Google Play Service API to a corresponding module, similar to what is currently done under UnifiedNlp Settings. Is this what you had in mind? Will some sort of module/plugin API be necessary?

It's still a bit unclear to me when something should be in its own module. Is this well defined or just when it seems intuitively appropriate enough (such as when the size becomes big)? I think having a list of all modules currently being considered (along with the reason for modularizing it) would help make sure everyone is on the same page for this plan.

Some potential downsides to consider:

1. Will some modules be required while others be optional? This may add confusion for users and require modifying the Self-Check UI and adding some good user documentation.
2. Will this mean each module will be versioned separately? This might make replicating user bug reports more complicated (e.g. figuring out which version of each module they're using).

This sounds like a significant organizational change both for development and for end user experience, so it might be worth doing as a 0.3 release (if this is considered a major/breaking change).

Also, as a side note, I think having signed nightlies would help reduce the number of duplicate bug reports. Oftentimes a release will introduce bugs/regressions that are quickly fixed but are not included in a release until months later. Users can't easily use it since self builds have a different signature.

[mar-v-in]

It could be nice to have a centralized Activity in microG that will show a mapping from each supported Google Play Service API to a corresponding module, similar to what is currently done under UnifiedNlp Settings. Is this what you had in mind? Will some sort of module/plugin API be necessary?

We can probably make use of dynamite, the dynamic module loading system Google designed for play services and which we have to implement anyway :) Some sort of activity that allows to select the desired module would still be necessary.

It's still a bit unclear to me when something should be in its own module. Is this well defined or just when it seems intuitively appropriate enough (such as when the size becomes big)? I think having a list of all modules currently being considered (along with the reason for modularizing it) would help make sure everyone is on the same page for this plan.

I guess the main reason to have modules as separate APK is when there is multiple choices or choices would make sense, like when there is different data sources. However, modules might also just be for code separation and there might be other usages.

1. Will some modules be required while others be optional? This may add confusion for users and require modifying the Self-Check UI and adding some good user documentation.

I was considering to replace the Self-Check with warnings and a setup assistant. Setup assistant would also help with people that install microG and then see it "not working" and just uninstall again - and it would help existing users with the transition to 0.3.

2. Will this mean each module will be versioned separately? This might make replicating user bug reports more complicated (e.g. figuring out which version of each module they're using).

I would still put the same version numbers on all modules as this makes everything easier. Also some changes might need to be reflected across multiple modules and it would just make sense if those will bump their version the same way.

This sounds like a significant organizational change both for development and for end user experience, so it might be worth doing as a 0.3 release (if this is considered a major/breaking change).

Yes, totally agree that this would be the reason to bump the version

Also, as a side note, I think having signed nightlies would help reduce the number of duplicate bug reports. Oftentimes a release will introduce bugs/regressions that are quickly fixed but are not included in a release until months later. Users can't easily use it since self builds have a different signature.

Yeah, I am trying to improve on that :)

[voidstarstar]

That sounds great!

I assume you're referring to dynamic delivery and app bundles. Do these features require F-Droid to support them? I can't seem to find any info on F-Droid support for the app bundle format, so it's unclear to me if this is currently possible.

[mar-v-in]

dynamite/chimera is a Google Play Service internal system with no documentation published yet (as far as I know). It allows to dynamically load and update .dex and resources without going through the package manager. It's not related to app bundles.

[ale5000-git]

1. Merge GmsApi, GmsLib and GmsCore: Very good idea, but I think pull requests should be merged before the change to avoid problems.

2. Upload remote-droid-guard library (*), wearable library and unifiednlp-base to maven central: Good idea.

3. Split play-services-core into smaller modules: Good idea.

4. Remove Android.mk build scripts from all repositories: If Android.mk isn't widely used and difficult to support then it is better to remove it but about an alternative repository with prebuilts is it really needed? There are already GitHub releases, F-Droid releases and microG download page releases (for microG GmsCore the last two are the same but for UnifiedNlp they are separate).

NOTE: I would like to insist on implementing the "reproducible builds", this allow to ship on the official F-Droid repository the apk files signed by you so they can be interchanged without the uninstallation.

5. Rename repositories to get rid of android_packages_ prefix...: Good idea, I also suggest to move legacy projects into a separate organization named, for example, "microg-superseded".

6. I propose, if possible, when you split maps api from the main GmsCore to make the main GmsCore again compatible with API 9 (if not possible to make it in a single apk maybe a separate flavour like the UnifiedNlp flavours), the next month I will buy an old phone for proper tests.

NOTE: A split in the apk could also be a good way to reduce memory usage by loading components only when they are needed.

7. I also propose to add a page inside microG settings that list all existing microG optional components and tell if they are installed or not (including remote-droid-guard and future map api v2 and also maps api v1) so the user know what is missing.

[mar-v-in]

@ale5000-git

make the main GmsCore again compatible with API 9

While I am all in for supporting older devices, I don't think supporting API 9 is really a good thing: Systems running API 9 usually have severe security issues, including known root exploits. While this is still fine for free software (because you can trust it for not making use of these exploits), it is not a good thing to run non-free stuff on such systems - and microG is mostly there to support non-free apps. Also, API 9 devices should probably be capped from internet and mobile network access for security reasons...

Also, supporting older API versions also means additional work in many cases and it's not like this project isn't enough work already 😉

That said, I will always try to keep the minSdkVersion at the lowest possible value, so the splitting up in components would probably make it easier to create a API 9 compatible version, even if it is not maintained by me anymore.

[ale5000-git]

@mar-v-in Certainly not useful for everyday use but you can reuse old phones just as music players (in this case security isn't needed at all) or security cams or other things and many apps, that maybe aren't open source but still may be free, need GmsCore.

The main use of GmsCore for old devices is using less resources and not crashing like the Google one ;)

Although having a separate version is feasible, it will complicate things. In the future I plan to test it on old devices and maybe also contributing but it will really be nice having the same codebase (maybe a separate flavour if needed).