Bootstrapping Bazel-6.0.0 from source in QEMU with RISCV image

[sainikhilrai]

Description of the bug:

I am trying to build the bazel from source on RISCV64 host device inside qemu. During compilation, I could see that compile.sh and bootstrap.sh scripts inside scripts/bootstrap/ directory got completed. Facing issue while running bazel_build command of compile.sh script. I could see that target was being build and throwing following error only when building //src/java_tools/buildjar/java/com/google/devtools/build/buildjar:starlark-deps and facing following error:

ERROR: /home/ubuntu/bazel-.0.0/src/java_tools/buildjar/java/com/google/devtools/build/buildjar/BUILD:169:23: JavacBootstrap src/java_tools/buildjar/java/com/google/devtools/build/buildjar/libstarlark-deps.jar [for tool] failed: (Exit 3): bash failed: error executing command (from target
 //src/java_tools/buildjar/java/com/google/devtools/build/buildjar:starlark-deps) 

After we get the above error, we encounter the following error:

The system is out of resources.
Consult the following stack trace for details.
java.lang.OutOfMemoryError: Java heap space
        at java.base/java.util.Spliterators.spliterator(Spliterators.java:178)
        at java.base/java.util.Arrays.spliterator(Arrays.java:5482)
        at java.base/java.util.Arrays.stream(Arrays.java:5633)
        at java.base/java.util.Arrays.stream(Arrays.java:5614)
        at java.base/java.util.stream.Stream.of(Stream.java:1188)

I even increased to HeapSize to 500G by using export BAZEL_JAVAC_OPTS="-J-Xmx500g" but still issue persists.

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

No response

Which operating system are you running Bazel on?

Linux ubuntu 6.2.0-19-generic #19.1-Ubuntu SMP Fri Mar 31 12:41:53 UTC 2023 riscv64 riscv64 riscv64 GNU/Linux

What is the output of bazel info release?

No response

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

1. prerequisite dependencies

sudo apt-get install build-essential openjdk-11-jdk python zip unzip

2. get bootstrapped distribution archive

wget https://github.com/bazelbuild/bazel/releases/download/6.0.0/bazel-6.0.0-dist.zip

3. Unzip the zip file

mkdir bazel-6.0.0 && unzip -d ./bazel-6.0.0 bazel-6.0.0-dist.zip && cd bazel-6.0.0

5. set the JAVA_HOME:

export JAVA_HOME="$(dirname $(dirname $(realpath $(which javac))))"

6. set the heap size to maximum to resolve HeapSize error:

export BAZEL_JAVAC_OPTS="-J-Xmx500g"

7. run the compile.sh

env EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" bash ./compile.sh

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

# Execution platform: //:default_host_platform                                                                                                  
INFO: From Compiling src/core/lib/channel/promise_based_filter.cc:                                                                              
In constructor 'grpc_core::promise_filter_detail::ServerCallData::PollContext::PollContext(grpc_core::promise_filter_detail::ServerCallData*, gr
pc_core::promise_filter_detail::BaseCallData::Flusher*)',                                                                                       
    inlined from 'void grpc_core::promise_filter_detail::ServerCallData::WakeInsideCombiner(grpc_core::promise_filter_detail::BaseCallData::Flus
her*)' at external/com_github_grpc_grpc/src/core/lib/channel/promise_based_filter.cc:1183:37:                                                   
external/com_github_grpc_grpc/src/core/lib/channel/promise_based_filter.cc:890:22: warning: storing the address of local variable 'poll_ctx' in 
'*this.grpc_core::promise_filter_detail::ServerCallData::poll_ctx_' [-Wdangling-pointer=]                                                       
  890 |     self_->poll_ctx_ = this;                                                                                                            
      |     ~~~~~~~~~~~~~~~~~^~~~~~                                                                                                             
external/com_github_grpc_grpc/src/core/lib/channel/promise_based_filter.cc: In member function 'void grpc_core::promise_filter_detail::ServerCal
lData::WakeInsideCombiner(grpc_core::promise_filter_detail::BaseCallData::Flusher*)':                                                           
external/com_github_grpc_grpc/src/core/lib/channel/promise_based_filter.cc:1183:15: note: 'poll_ctx' declared here                              
 1183 |   PollContext poll_ctx(this, flusher);                                                                                                  
      |               ^~~~~~~~                                                                                                                  
external/com_github_grpc_grpc/src/core/lib/channel/promise_based_filter.cc:1183:15: note: '<unknown>' declared here                             
SUBCOMMAND: # @com_github_grpc_grpc//:grpc_base [action 'Linking external/com_github_grpc_grpc/libgrpc_base.a', configuration: 2752f6466326ffa9a
1e16b67062896a2c767d69a70c4fe03239d5bcfe1259886, execution platform: //:default_host_platform]                                                  
(cd /tmp/bazel_yR4anOqC/out/execroot/io_bazel && \                                                                                              
  exec env - \                                                                                                                                  
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \                                                               
    PWD=/proc/self/cwd \                                                                                                                        
  /usr/bin/ar @bazel-out/riscv64-opt/bin/external/com_github_grpc_grpc/libgrpc_base.a-2.params)                                                 
# Configuration: 2752f6466326ffa9a1e16b67062896a2c767d69a70c4fe03239d5bcfe1259886                                                               
# Execution platform: //:default_host_platform                                                                                                  
SUBCOMMAND: # //src/main/cpp:client [action 'Linking src/main/cpp/client', configuration: 2752f6466326ffa9a1e16b67062896a2c767d69a70c4fe03239d5b
cfe1259886, execution platform: //:default_host_platform]   
(cd /tmp/bazel_yR4anOqC/out/execroot/io_bazel && \
  exec env - \
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \
    PWD=/proc/self/cwd \
  /usr/bin/gcc @bazel-out/riscv64-opt/bin/src/main/cpp/client-2.params)
# Configuration: 2752f6466326ffa9a1e16b67062896a2c767d69a70c4fe03239d5bcfe1259886
# Execution platform: //:default_host_platform
ERROR: /home/ubuntu/bazel-6.0.0/src/java_tools/buildjar/java/com/google/devtools/build/buildjar/BUILD:169:23: JavacBootstrap src/java_tools/buil
djar/java/com/google/devtools/build/buildjar/libstarlark-deps.jar [for tool] failed: (Exit 3): bash failed: error executing command (from target
 //src/java_tools/buildjar/java/com/google/devtools/build/buildjar:starlark-deps)
  (cd /tmp/bazel_yR4anOqC/out/execroot/io_bazel && \
  exec env - \
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \
  /bin/bash -c 'set -e;rm -rf bazel-out/riscv64-opt-exec-EDC14992/bin/src/java_tools/buildjar/java/com/google/devtools/build/buildjar/libstarlar
k-deps.jar.build_output bazel-out/riscv64-opt-exec-EDC14992/bin/src/java_tools/buildjar/java/com/google/devtools/build/buildjar/libstarlark-deps
.jar.build_java bazel-out/riscv64-opt-exec-EDC14992/bin/src/java_tools/buildjar/java/com/google/devtools/build/buildjar/libstarlark-deps.jar.bui
ld_java_list

[sainikhilrai]

Able to build the binaries successfully now. Hence closing the issue.

[bodhi-mnagarajan]

Reported issue occurs because java max heap size passed through BAZEL_JAVAC_OPTS="-J-Xmx500g" is not used during the javac command execution from tools/build_rules/java_rules_skylark.bzl (_java_library_impl). During this javac command execution, java heap size is likely selected based on system configuration and runs out of memory.

Not sure how to pass user defined heap size parameter through environmental variable or command line options in Bazel build through bootstrapping. Tried few options as below to pass user defined java heap size configuration but these ended up with different build failures.

a) env EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk --host_javacopt=-J-Xmx200g" bash ./compile.sh

ERROR: /local/mnagarajan/bazel-bootstrap-src/src/main/java/com/google/devtools/build/lib/windows/BUILD:24:13: Building src/main/java/com/google/devtools/build/lib/windows/libwindows_short_path.jar (1 source file) failed: Worker process did not return a WorkResponse:

---8<---8<--- Start of log, file at /tmp/bazel_uCp602yD/out/bazel-workers/multiplex-worker-1-Javac.log ---8<---8<---
Exception in thread "main" java.lang.IllegalArgumentException: error: invalid flag: -J-Xmx200g
	at jdk.compiler/com.sun.tools.javac.main.Arguments.reportDiag(Arguments.java:891)
	at jdk.compiler/com.sun.tools.javac.main.Arguments.doProcessArgs(Arguments.java:395)
	at jdk.compiler/com.sun.tools.javac.main.Arguments.processArgs(Arguments.java:347)
	at jdk.compiler/com.sun.tools.javac.main.Arguments.init(Arguments.java:246)
	at jdk.compiler/com.sun.tools.javac.api.JavacTool.getTask(JavacTool.java:185)
	at jdk.compiler/com.sun.tools.javac.api.JavacTool.getTask(JavacTool.java:119)
	at jdk.compiler/com.sun.tools.javac.api.JavacTool.getTask(JavacTool.java:68)
	at com.google.devtools.build.buildjar.VanillaJavaBuilder.run(VanillaJavaBuilder.java:175)
	at com.google.devtools.build.buildjar.VanillaJavaBuilder.runPersistentWorker(VanillaJavaBuilder.java:106)
	at com.google.devtools.build.buildjar.VanillaJavaBuilder.main(VanillaJavaBuilder.java:87)

b) env EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk --javacopt=-J-Xmx200g" bash ./compile.sh

=> again parameters passed using --javacopt are not picked up

However if tools/build_rules/java_rules_skylark.bzl (_java_library_impl) is directly patched up to take this java heap size parameter, Bazel build succeeds and Bazel binary is generated successfully.

bazel-6.1.1-dist/tools/build_rules/java_rules_skylark.bzl
@@ -68,7 +68,6 @@ def _java_library_impl(ctx):
     if ctx.files.srcs or ctx.files.srcjars:
         cmd += "%s/bin/javac" % java_runtime.java_home
         cmd += " " + " ".join(javac_options)
-        cmd += " -J-Xmx200g "
         if compile_time_jars:

@https://github.com/bazelbuild/bazel/labels/team-OSS pls suggest the right way to solve this

[meteorcloudy]

@cushon @hvadehra any idea here?

[cushon]

If --host_javacopt=-J-Xmx200g is passed, those options are passed to both the bootstrap ./tools/build_rules/java_rules_skylark.bzl rules, and the regular java_library implementation. The latter doesn't support -J flags.

[bodhi-mnagarajan]

@cushon Is there a way to pass those options only to the bootstrap ./tools/build_rules/java_rules_skylark.bzl rules ?

[shata-wh]

Able to build the binaries successfully now. Hence closing the issue.

How did you build it successfully? I tried to bootstrap bazel-3.1.0 on riscv, and I also encountered this problem.

[root@openeuler-riscv64 bazel-3.1.0]# env EXTRA_BAZEL_ARGS="--host_javabase=@local_jdk//:jdk" bash ./compile.sh
🍃  Building Bazel from scratch../usr/lib/jvm/java-1.8.0-openjdk-1.8.0.352.b08-5.oe2303.riscv64/bin/javac -classpath derived/jars/com_google_protobuf/libprotobuf_java.jar:derived/jars/com_google_protobuf/libprotobuf_java_util.jar:third_party/truth/truth-1.0.1.jar:third_party/tomcat_annotations_api/tomcat-annotations-api-8.0.5.jar:third_party/apache_commons_lang/commons-lang-2.6.jar:third_party/opencensus/opencensus-api-0.19.2.jar:third_party/opencensus/opencensus-contrib-grpc-metrics-0.19.2.jar:third_party/hamcrest/hamcrest-core-1.3.jar:third_party/bytebuddy/byte-buddy-1.9.7.jar:third_party/bytebuddy/byte-buddy-agent-1.9.7.jar:third_party/asm/asm-analysis-7.0-sources.jar:third_party/asm/asm-7.0.jar:third_party/asm/asm-7.0-sources.jar:third_party/asm/asm-util-7.0-sources.jar:third_party/asm/asm-util-7.0.jar:third_party/asm/asm-commons-7.0.jar:third_party/asm/asm-commons-7.0-sources.jar:third_party/asm/asm-tree-7.0.jar:third_party/asm/asm-tree-7.0-sources.jar:third_party/asm/asm-analysis-7.0.jar:third_party/apache_velocity/velocity-1.7.jar:third_party/gson/gson-2.2.4.jar:third_party/hungarian_algorithm/software-and-algorithms-1.0-src.jar:third_party/hungarian_algorithm/software-and-algorithms-1.0.jar:third_party/auth/google-auth-library-oauth2-http-0.17.1.jar:third_party/auth/google-auth-library-credentials-0.17.1.jar:third_party/instrumentation/instrumentation-api-0.4.3.jar:third_party/android_common/com.android.tools.build_builder-model_2.0.0.jar:third_party/android_common/com.android.tools.layoutlib_layoutlib_26.1.2.jar:third_party/android_common/com.android.tools_sdklib_25.0.0.jar:third_party/android_common/com.android.tools.layoutlib_layoutlib_26.1.2-stripped.jar:third_party/android_common/com.android.tools.build_builder-test-api_2.0.0.jar:third_party/android_common/com.android.tools_repository_25.0.0.jar:third_party/android_common/com.android.tools.build_manifest-merger_25.0.0-patched.jar:third_party/android_common/com.android.tools_sdk-common_25.0.0.jar:third_party/android_common/com.android.tools_sdk-common_25.0.0-stripped.jar:third_party/android_common/com.android.tools.lint_lint-checks_25.0.0.jar:third_party/android_common/com.android.tools.lint_lint-api_25.0.0.jar:third_party/android_common/com.android.tools.external.lombok_lombok-ast_0.2.3.jar:third_party/android_common/com.android.tools_common_25.0.0.jar:third_party/android_common/com.android.tools.build_builder_2.0.0.jar:third_party/android_common/com.android.tools_dvlib_25.0.0.jar:third_party/android_common/com.android_annotations_25.0.0.jar:third_party/allocation_instrumenter/java-allocation-instrumenter-3.0.1.jar:third_party/pcollections/pcollections-2.1.2.jar:third_party/objenesis/objenesis-1_3.jar:third_party/java/jdk/langtools/java_compiler-src.jar:third_party/java/jdk/langtools/jdk_compiler-src.jar:third_party/java/jdk/langtools/jdk_compiler.jar:third_party/java/jdk/langtools/java_compiler.jar:third_party/java/jdk/langtools/javac-9+181-r4173-1.jar:third_party/java/javapoet/javapoet-1.8.0.jar:third_party/java/android_databinding/v2_3_1/exec.jar:third_party/java/jcommander/jcommander-1.48.jar:third_party/java/proguard/proguard5.3.3/examples/annotations/lib/annotations.jar:third_party/java/proguard/proguard5.3.3/lib/retrace.jar:third_party/java/proguard/proguard5.3.3/lib/proguardgui.jar:third_party/java/proguard/proguard5.3.3/lib/proguard.jar:third_party/java/jacoco/org.jacoco.agent-0.8.3-sources.jar:third_party/java/jacoco/jacocoagent-0.8.3.jar:third_party/java/jacoco/org.jacoco.core-0.8.3.jar:third_party/java/jacoco/org.jacoco.report-0.8.3.jar:third_party/java/jacoco/org.jacoco.report-0.8.3-sources.jar:third_party/java/jacoco/org.jacoco.core-0.8.3-sources.jar:third_party/java/jacoco/org.jacoco.agent-0.8.3.jar:third_party/apache_commons_logging/commons-logging-1.1.1.jar:third_party/xz/xz-1.5.jar:third_party/jaxb/jaxb-api-2.3.1-patched.jar:third_party/jaxb/jaxb-api-2.3.1.jar:third_party/jaxb/jaxb-api-2.3.1-sources.jar:third_party/turbine/turbine_direct.jar:third_party/netty_tcnative/netty-tcnative-boringssl-static-2.0.24.Final.jar:third_party/truth8/truth-java8-extension-1.0.1.jar:third_party/jcip_annotations/jcip-annotations-1.0-1.jar:third_party/netty/netty-all-4.1.34.Final.jar:third_party/diffutils/diffutils-1.3.0.jar:third_party/mockito/mockito-core-2.25.1.jar:third_party/flogger/flogger-0.5.1.jar:third_party/flogger/google-extensions-0.5.1.jar:third_party/flogger/flogger-system-backend-0.5.1.jar:third_party/jimfs/jimfs-1.1.jar:third_party/jsr305/jsr-305.jar:third_party/jackson2/jackson-core-2.8.6.jar:third_party/checker_framework_dataflow/dataflow-2.5.3.jar:third_party/checker_framework_dataflow/dataflow-2.5.3-sources.jar:third_party/auto/auto-value-1.6.3rc1.jar:third_party/auto/auto-value-annotations-1.6.3rc1.jar:third_party/auto/auto-common-0.10.jar:third_party/auto/auto-service-1.0-rc4.jar:third_party/error_prone/error_prone_type_annotations-2.3.2-SNAPSHOT.jar:third_party/error_prone/error_prone_annotations-2.2.0.jar:third_party/error_prone/error_prone_annotation-2.3.2-SNAPSHOT.jar:third_party/guava/guava-25.1-jre.jar:third_party/error_prone/error_prone_check_api-2.3.2-SNAPSHOT.jar:third_party/javax_annotations/javax.annotation-api-1.3.2-sources.jar:third_party/javax_annotations/javax.annotation-api-1.3.2.jar:third_party/ijar/test/libwrongcentraldir.jar:third_party/ijar/test/jar-with-manifest-and-target-label.jar:third_party/ijar/test/jar-with-manifest.jar:third_party/ijar/test/jar-without-manifest.jar:third_party/ijar/test/nestmates/nestmates.jar:third_party/jsr330_inject/javax.inject.jar:third_party/compile_testing/compile-testing-0.18.jar:third_party/apache_commons_pool2/commons-pool2-2.3.jar:third_party/api_client/google-api-client-1.22.0.jar:third_party/api_client/google-http-client-1.22.0-SNAPSHOT.jar:third_party/api_client/google-api-client-jackson2-1.22.0.jar:third_party/api_client/google-http-client-jackson2-1.22.0.jar:third_party/apache_commons_compress/apache-commons-compress-1.9.jar:third_party/checker_framework_annotations/checker-qual-3.0.0-sources.jar:third_party/checker_framework_annotations/checker-qual-3.0.0.jar:third_party/junit/junit-4.13.jar:third_party/bazel_bootstrap/libautocodec-annotation.jar:third_party/bazel_bootstrap/libautocodec-processor.jar:third_party/bazel_bootstrap/libserialization.jar:third_party/bazel_bootstrap/libserialization-processor-util.jar:third_party/bazel_bootstrap/libserialization-constant-processor.jar:third_party/bazel_bootstrap/libserialization-constant-annotation.jar:third_party/bazel_bootstrap/libregistered-singleton.jar:third_party/bazel_bootstrap/libunsafe-provider.jar:third_party/apache_commons_collections/commons-collections-3.2.2.jar:third_party/checker_framework_javacutil/javacutil-2.5.3-sources.jar:third_party/checker_framework_javacutil/javacutil-2.5.3.jar:third_party/grpc/grpc-netty-1.20.0.jar:third_party/grpc/grpc-stub-1.20.0.jar:third_party/grpc/grpc-auth-1.20.0.jar:third_party/grpc/grpc-protobuf-lite-1.20.0.jar:third_party/grpc/grpc-core-1.20.0.jar:third_party/grpc/grpc-context-1.20.0.jar:third_party/grpc/grpc-protobuf-1.20.0.jar:third_party/guava/guava-testlib-25.1-jre.jar:third_party/error_prone/error_prone_core-2.3.2-SNAPSHOT.jar:/tmp/bazel_WtHciobP -sourcepath src/java_tools/singlejar/java/com/google/devtools/build/zip:src/main/java:tools/java/runfiles:third_party/java/dd_plist/java:/tmp/bazel_WtHciobP/src -d /tmp/bazel_WtHciobP/classes -source 1.8 -target 1.8 -encoding UTF-8 @/tmp/bazel_xMOU5xqc/param


The system is out of resources.
Consult the following stack trace for details.
java.lang.OutOfMemoryError: Java heap space
        at com.sun.tools.javac.util.Position$LineMapImpl.build(Position.java:153)
        at com.sun.tools.javac.util.Position.makeLineMap(Position.java:77)
        at com.sun.tools.javac.parser.JavadocTokenizer.getLineMap(JavadocTokenizer.java:445)
        at com.sun.tools.javac.parser.Scanner.getLineMap(Scanner.java:127)
        at com.sun.tools.javac.parser.JavacParser.parseCompilationUnit(JavacParser.java:3173)
        at com.sun.tools.javac.main.JavaCompiler.parse(JavaCompiler.java:628)
        at com.sun.tools.javac.main.JavaCompiler.parse(JavaCompiler.java:665)
        at com.sun.tools.javac.main.JavaCompiler.parseFiles(JavaCompiler.java:950)
        at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:857)
        at com.sun.tools.javac.main.Main.compile(Main.java:523)
        at com.sun.tools.javac.main.Main.compile(Main.java:381)
        at com.sun.tools.javac.main.Main.compile(Main.java:370)
        at com.sun.tools.javac.main.Main.compile(Main.java:361)
        at com.sun.tools.javac.Main.compile(Main.java:56)
        at com.sun.tools.javac.Main.main(Main.java:42)

Thank you.