Bazel OOMs during analysis phase when KSP provides Java outputs information

[EdbertChan]

When invoking kt_compiler_deps_aspect_impl on a large enough build graph, output infos will be merged and Bazel will throw an out of memory error.

The stack trace:

net.starlark.java.eval.Starlark$UncheckedEvalError: OutOfMemoryError thrown during Starlark evaluation (//libraries/foundation/analytics/jvm:src_main)
	at <starlark>.merge(<builtin>:0)
	at <starlark>.kt_compiler_deps_aspect_impl(/Users/edbert/.bazelcache/workspace/ff080e9ef5254374b7ce1c9f144b976d/external/io_bazel_rules_kotlin/kotlin/internal/jvm/impl.bzl:362)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at java.base/java.util.Arrays.copyOf(Unknown Source)
	at com.google.common.collect.ImmutableList$Builder.getReadyToExpandTo(ImmutableList.java:795)
	at com.google.common.collect.ImmutableList$Builder.add(ImmutableList.java:814)
	at com.google.devtools.build.lib.rules.java.JavaRuleOutputJarsProvider$Builder.addJavaOutput(JavaRuleOutputJarsProvider.java:284)
	at com.google.devtools.build.lib.rules.java.JavaRuleOutputJarsProvider.merge(JavaRuleOutputJarsProvider.java:261)
	at com.google.devtools.build.lib.rules.java.JavaInfo.merge(JavaInfo.java:158)
	at com.google.devtools.build.lib.rules.java.JavaStarlarkCommon.mergeJavaProviders(JavaStarlarkCommon.java:247)
	at jdk.internal.reflect.GeneratedMethodAccessor263.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at net.starlark.java.eval.MethodDescriptor.call(MethodDescriptor.java:162)

This is because kt_compiler_deps_aspect_impl invokes java_common.merge(infos) which naively merges all the information. The OOM kicks in because Bazel is trying to copy over 500M transitive outputs.

The easiest way to reproduce this is to create a large kotlin project that has a lot of dependencies that have multiple paths between the root project and the leaves.

This is similar to Ben's issue for Java Plugins. We want similar capabilities where KSP can hook into the JavaPluginInfo. Unfortunately, java_common does not give us this out of the box.

[restingbull]

TODO:

• do we still need the aspect?
• Clean up provider structure.