ResourceProcessorBusyBox crashes with a lot of assets

[kitterion]

Description of the problem / feature request:

ResourceProcessorBusyBox crashes with Argument list too long if android_{binary,library} has a lot of assets, even if they are all in the same folder:

Exception in thread "main" com.google.devtools.build.android.aapt2.ResourceLinker$LinkError: java.io.IOException: Cannot run program "bazel-out/k8-opt-exec-2B5CBBC6/bin/external/androidsdk/aapt2_binary": error=7, Argument list too long
        at com.google.devtools.build.android.aapt2.ResourceLinker.link(ResourceLinker.java:561)
        at com.google.devtools.build.android.Aapt2ResourcePackagingAction.main(Aapt2ResourcePackagingAction.java:471)
        at com.google.devtools.build.android.ResourceProcessorBusyBox$Tool$8.call(ResourceProcessorBusyBox.java:111)
        at com.google.devtools.build.android.ResourceProcessorBusyBox.processRequest(ResourceProcessorBusyBox.java:234)
        at com.google.devtools.build.android.ResourceProcessorBusyBox.main(ResourceProcessorBusyBox.java:177)
Caused by: java.io.IOException: Cannot run program "bazel-out/k8-opt-exec-2B5CBBC6/bin/external/androidsdk/aapt2_binary": error=7, Argument list too long
        at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
        at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
        at com.google.devtools.build.android.CommandHelper.execute(CommandHelper.java:29)
        at com.google.devtools.build.android.AaptCommandBuilder.execute(AaptCommandBuilder.java:297)
        at com.google.devtools.build.android.aapt2.ResourceLinker.linkProtoApk(ResourceLinker.java:432)
        at com.google.devtools.build.android.aapt2.ResourceLinker.link(ResourceLinker.java:544)
        ... 4 more
Caused by: java.io.IOException: error=7, Argument list too long
        at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
        at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:340)
        at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:271)
        at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
        ... 9 more

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Having this code in a BUILD.bazel file and trying to build bin, ResourceProcessorBusyBox throws the exception mentioned above

LONG_PATH = "a" * 200
RANGE = [str(i) for i in range(1, 5000)]

android_binary(
    name = "bin",
    manifest = "AndroidManifest.xml",
    assets = [":file" + i for i in RANGE],
    assets_dir = LONG_PATH,
)

[genrule(
    name = "file" + i,
    outs = [LONG_PATH + "/" + i],
    cmd = "touch $@",
) for i in RANGE]

What operating system are you running Bazel on?

Linux

What's the output of bazel info release?

release 4.0.0

Have you found anything relevant by searching the web?

No

Any other information, logs, or outputs that you want to share?

This is especially painful on MacOS, where the command line limit is 256KiB, 8 times lower than on any linux machine I could find, and so this limit is hit way sooner.

I have a fix for this crash here https://github.com/kitterion/bazel/commit/1ca0724bbb1348208da491cd61403d3d38dab399. While it does fix the problem and I could send it as a PR, I'm not sure if this is the correct solution.

Pre-alpha rules_android already has similar code https://github.com/bazelbuild/rules_android/blob/87690250441364c8dfab01b2f4ee6fe67a44277a/rules/busybox.bzl#L32-L70 that deduplicates asset dirs, it would be nice to have it in native rules until the starlark rewrite is ready to replace them.

[kkpattern]

Hi, is it possible for the fix https://github.com/kitterion/bazel/commit/1ca0724bbb1348208da491cd61403d3d38dab399 to be merged? Without this fix, it costs 527 seconds to process all the resources in our android application. With the patch applied only 160 seconds.

We tested the patch with 5.3.0 and it's still valid, if @kitterion is busy we can help prepare the PR.

Thanks.

[kitterion]

My bad, I should've sent it long ago. I'll do it shortly. Thanks for the ping.

[kkpattern]

My bad, I should've sent it long ago. I'll do it shortly. Thanks for the ping.

Not at all. Thanks so much for your fix!