AndroidX considering dropping support for .pom files

[liutikas]

AndroidX is considering dropping support for .pom files and just using .module. .pom files do not really work with KMP, where there is no such thing as a single artifact, it all depends on the consumer, e.g. if the consumer if JVM vs Android you get a different artifact. AndroidX KMP usage is growing, and we keep putting in really ugly workarounds in our build to keep .pom files barely usable (e.g. https://github.com/bazel-contrib/rules_jvm_external/issues/864). The maintenance burden has really reached a tipping point.

This change would be a no-op for our Gradle users, but likely breaking for alternative build system users (e.g Bazel, Buck, etc).

I can see that Bazel is looking to support Gradle resolver (https://github.com/bazel-contrib/rules_jvm_external/pull/1357) instead of using poms. This change would be required for all the bazel users to keep consuming AndroidX artifacts.

We would obviously give an x month heads up to the world when we do this change.

I wanted to create this issue to start the discussion.

[shs96c]

Thanks for starting the conversation. It's appreciated.

I have a number of concerns, and I think these need to be addressed before pom support is dropped by AndroidX.

1. How will cross-platform lock files be generated? A persistent problem with (eg) the Python rules is that generally the tools being used to generate the lock files don't handle the cross-platform cases (that is, if the dependency resolution is done on macOS, the dependencies from Linux or Windows aren't collected). Since a common use-case for Bazel involves cross-platform builds (eg. macOS client, and something like EngFlow as the RBE) this won't work. Further Open Source projects using Bazel need to work on multiple different Operating Systems, and should also have lock files that work "out of the box" for people wanting to contribute.
2. The Gradle resolver we're landing is still experimental, and it's significantly slower than either the Maven or coursier-backed solutions. For large repos, it's currently not practical to pin using it. This will need to be addressed somehow.
3. pom files are part of the glue that hold the JVM-based language ecosystems together by allowing people to choose the tool that works best for them. Effectively forcing everyone on to Gradle doesn't seem like a healthy thing to do for the wider community. Presumably, the AndroidX folks are also working with (at least!) the Maven team (and us!) to ensure that alternative build tooling continues to work?

The first problem could be addressed by having a lock file per language, but the files are large, prone to changing often, and this solution would likely require all users to have access to macOS, Linux, and Windows machines, and that's just not practical. A single multi-platform lock file is desirable How are AndroidX/Gradle planning to resolve this problem?

The second problem will require a significant lead-time. The main contributors to rules_jvm_external do so in their free time, and that obviously impacts development speed :) We could move faster with help from someone who knows the internals of Gradle very well in order to ensure the new gradle resolver works on large, cross-platform repos. Do you have someone on your team who'd be willing to help?

[shs96c]

@liutikas BTW, I'm not sure whether you're part of the AndroidX team or just passing on some news. If it's the latter, my apologies for the requests for help.

[liutikas]

I am not sure I can answer the questions you raised. In particular 1 and 2.

For 3, pom are held by ducttape and glue in terms of publishing from artifacts Gradle. AndroidX continues to spend a lot of effort to keep it barely functioning, so it is hard to call it the glue that works best. This is especially true for Android and KMP libraries.

Here are some examples of such issues:

1. pom spec requires the user of a dependency to specify <type>aar</type> in the <dependencies> block if that dependency is an android library, otherwise the default is jar and many maven resolvers will fail. Gradle does not add these type entries, so AndroidX has a workaround in our build add these. We can only do that for our own libraries as we have a list we can reference, so if AndroidX depends on an non-androidx library that is an AAR, pom files generated are already broken. In Gradle metadata, this is not an issue, as the types of artifacts are described in the actual dependency's metadata.
2. Related to issue 1 above, let's look at a case where an artifact com.foo:bar:1.0.0 migrates from an android library (AAR) to a KMP library that also supports Android in version 1.1.0. An existing other library that depends on com.foo:bar:1.0.0 will have <type>aar</type> and if the project pulls in this other library and com.foo:bar:1.1.0 this breaks many maven resolvers as it tries to look for an aar under com/foo/bar/1.1.0 which no longer exists there, it is now under com/foo/bar-android/1.1.0. Issue tracking this. To work around it in androidx we put a fake aar under com/foo/bar/1.1.0 to keep pom users working. While this fixes the AAR resolution, it actually drops Kotlin metadata jar files, making kotlinc miss Protos based metadata stored there.
3. KMP has no concept of "default platform" for pom files. A vanilla KMP project that has android and jvm variants will have an empty pom file. Combined with issue 2 above, this means that maven resolvers using pom files do not actually either jar or aar. To work around it in androidx build we put a hack that picks a default for users by adding an extra dependency to com.foo:bar-android or com.foo:bar-jvm based on default is reasonable for this artifact.
4. Because of the flattening in issue 3 above, we can only pick either JVM or Android version of artifacts for a given artifact. This leads us to issues where the wrong variant is picked for either android or JVM consumers. Let's say we have "com.fun:fun" (default JVM) and "com.lol:lol" (default android). If com.lol:lol depends on com.fun:fun, using pom resolution, you'll get JVM version of com.fun:fun despite the android variant being available, which means there might be calls to java.* types that simply don't exist on android.
5. When generating pom files for KMP, Kotlin Gradle Plugin has to pick dependencies for each artifact variant, when it does it can do anchor com.foo:foo or com.foo:foo-jvm and today it does it inconsistently, AndroidX has workaround to undo some of these as well.

AndroidX has always only officially supported Gradle users, other consumers have been best-effort. We are definitely not staffed to solve lock file issues or build Gradle bridges, as we have other pressing issues for our majority Gradle users.

I recognize this puts bazel and other build systems in a tough spot.

[martinbonnin]

Been thinking a bit more about this and I believe at its core, it's a graph modelisation problem.

With .pom files, different targets can be modelled as different subtrees. This works well but requires the consumer to make an explicit choice about the target they want to support:

---
title: .pom files
---

%%{init: {"theme": "dark" }}%%
graph TB
    app --explicit choice--> lib1-android
    app -. unused .-> lib1-jvm;
    subgraph  
    direction TB
    lib1-jvm --> lib2-jvm
    end
    lib1-android --> lib2-android

    classDef android color:#000,fill:#a4c639
    lib1-android:::android
    lib2-android:::android

.module files on the other hand, allow to group "similar" artifacts together so that the the consumer can use "short-names":

(resolved artifacts in green)

---
title: .module files
---

%%{init: {"theme": "dark" }}%%
graph TB
   app --> lib1
   subgraph one[lib1]
   lib1 --> lib1-android
   lib1 --> lib1-jvm
   end 
   lib1 --> lib2[lib2]
   subgraph two
   lib2 --> lib2-android
   lib2 --> lib2-jvm
   end

   classDef android color:#000,fill:#a4c639
   lib1-android:::android
   lib2-android:::android
   lib1:::android
   lib2:::android

The core of the issue seems to be that we want to use "short names" that are the same irrespective of the actual target, which is obviously not possible using .pom files?

But is that so much of an issue?

Having separate artifact names for different targets (a.k.a. flavours/variants) of a library seems ok-ish to me.

Don't get me wrong, I'd welcome improvements to maven dependency resolution algorithms but dropping .pom files feels like a big irreversible step and I'm not sure Gradle .module files in their current form are the answer to this question.

Supporting Gradle metadata is a lot of effort accross the whole ecosystem as the Bazel PR shows (if even it is possible considering any Gradle plugin can influence the resolution with disambiguation rules).

Maybe the resolution rules can be made simpler and a few core attributes made immutable so that disambiguation rules would not have to be duplicated accross every build tool?

[liutikas]

Having separate artifact names for different targets (a.k.a. flavours/variants) of a library seems ok-ish to me.

Different artifacts work only if every participant down the dependency chain properly set up the targets. It is still possible to accidentally get both -android and -jvm from your example if some library decided they worked on an artifact just for JVM and that gets pulled in through transitive dependencies.

Additionally, any lower level library is unable to split into multiple artifacts without breaking all the depend libraries as that forces all of those libraries to re-release. I assume you are well aware that there are many useful feature complete libraries that would likely never see a re-release.

Finally, it is relatively easy when you only have android/JVM (two way split), but as soon as you get to combination of OS/arch/java-version it is a combinatoric explosion that has to be manually curated.

[martinbonnin]

@liutikas agreed with all your points but still concerned that we are replacing one evil with another one. All of that while breaking non-Gradle consumers at the same time...

.pom has its limitations but dropping the format in favor of .module files and leaving Bazel and other build systems (and Kotlin scripting!) on the side of the road doesn't feel great.

Feels like AndroidX, as a large contributor of the ecoystem is in a unique position to influence the future of the metadata format.

@liutikas How doable would it be for AndroidX to commit to a subset of .module that is reasonnably possible to implement without having to embed Gradle?

@shs96c I see https://github.com/bazel-contrib/rules_jvm_external/pull/1357 has landed. Did you document what led you to the conclusion that embedding Gradle was the good tradeoff there? Is there anything that can be done to the .module format so that it could be implemented without having to spawn a full Gradle instance?