Bazel crash (no error) with env HOME=not/a/directory

[c-parsons]

HOME=not/a/directory bazel --output_user_root=bogus info used to succeed in giving informative output. However, as of a few weeks ago, this causes Bazel server to crash with no error.

$ HOME=not/a/directory bazel --output_user_root=bogus info
Starting local Bazel server and connecting to it...
<server dies, returning failure>
$

While this seems like a convoluted test case, this is actually something which hit the effort to move Bazel into AOSP. (We were incorrectly specifying HOME environment, alongside other flags, and Bazel used to not care. But now Bazel crashes!)

According to git bisect, this breakage was caused by https://github.com/bazelbuild/bazel/commit/86451bb2cfd360dc743cd0c2839063459c2600f4, which I believe you were a reviewer for, @Wyverald.

[c-parsons]

(Also re: reproducing this error, the --output_user_root=bogus is also important, because otherwise bazel crashes with a descriptive error. The error is only suppressed if both HOME and --output_user_root are specified)

[Wyverald]

Is this a P1? Can you specify a good HOME env var?

[c-parsons]

It should be P1 because this is a recent regression that results in a crash bug.

IIRC we have a Bazel policy that crash bugs are P1 by default, and this is a pretty difficult-to-diagnose server crash.

In terms of the effort to move Bazel into AOSP, we can indeed fix our HOME path, but that doesn't change the argument for making this P1.

[Wyverald]

Not that I don't believe you, but if you could link to the policy, that'd be great.

I was hoping this could be downgraded since it appears hard to trigger and we already have a crapton of P1 issues to fix before the 6.0 cut. Meanwhile, I'll keep this one in mind.

[c-parsons]

No, you're right to question me making an assertion about policy with "IIRC". Frankly, I can't find the policy, so I'm not sure if I accidentally made it up. If so, my bad.

I still think we should make a case to have this be a P1, because a server crash with no error info is not something I think we want in our Bazel 6.0 release. FYI @lberki

[lberki]

@c-parsons emotionally, I feel similarly, but I think a better policy is that a new server crash with no error info should not be in Bazel 6.0; the distinction is important because the opposite would imply that we have to fix all such crashes before Bazel 6.0 can be released.

So I'll thereby add this bug to the hotlist of release blockers unless @Wyverald has some really great arguments why that should not be the case.

I also started a thread about what our policy wrt. crash bugs should be like.

[meteorcloudy]

I'm looking into this one, it seems the problem isn't because setting HOME to an no-exist directory, but because it was set to a relative path (which is also weird for HOME).

pcloudy@pcloudy-macbookpro3:~/workspace/my_tests/bazel (master)
$ mkdir foo
pcloudy@pcloudy-macbookpro3:~/workspace/my_tests/bazel (master)
$ touch foo/.netrc
pcloudy@pcloudy-macbookpro3:~/workspace/my_tests/bazel (master)
$ HOME=./foo bazel  --client_debug info
2022/09/27 15:41:00 Using unreleased version at commit 59e51fe625f485dc664928ae0ad93035808779f3
2022/09/27 15:41:00 Downloading https://storage.googleapis.com/bazel-builds/artifacts/macos_arm64/59e51fe625f485dc664928ae0ad93035808779f3/bazel...
2022/09/27 15:41:00 Skipping basic authentication for storage.googleapis.com because no credentials found in foo/.netrc
[INFO 15:41:02.984 src/main/cpp/option_processor.cc:407] Looking for the following rc files: /etc/bazel.bazelrc,/Users/pcloudy/workspace/my_tests/bazel/.bazelrc,./foo/.bazelrc
[INFO 15:41:02.984 src/main/cpp/rc_file.cc:56] Parsing the RcFile /Users/pcloudy/workspace/my_tests/bazel/.bazelrc
[INFO 15:41:02.985 src/main/cpp/rc_file.cc:56] Parsing the RcFile user.bazelrc
[INFO 15:41:02.985 src/main/cpp/rc_file.cc:131] Skipped optional import of user.bazelrc, the specified rc file either does not exist or is not readable.
[INFO 15:41:02.985 src/main/cpp/blaze.cc:1697] Debug logging requested, sending all client log statements to stderr
[INFO 15:41:03.012 src/main/cpp/blaze.cc:1575] Acquired the client lock, waited 0 milliseconds
[USER 15:41:03.027 src/main/cpp/blaze.cc:885] Starting local Bazel server and connecting to it...
[INFO 15:41:04.066 src/main/cpp/blaze.cc:1773] Trying to connect to server (timeout: 30 secs)...
[INFO 15:41:04.120 src/main/cpp/blaze.cc:1269] Connected (server pid=65512).
[INFO 15:41:04.120 src/main/cpp/blaze.cc:2049] Releasing client lock, let the server manage concurrent requests.
[INFO 15:41:04.730 src/main/cpp/blaze.cc:2168] failure_detail: message: "Crashed: (java.lang.IllegalArgumentException) Paths must be absolute: \'foo/.netrc\'"
crash {
  causes {
    throwable_class: "java.lang.IllegalArgumentException"
    message: "Paths must be absolute: \'foo/.netrc\'"
    stack_trace: "com.google.common.base.Preconditions.checkArgument(Preconditions.java:220)"
    stack_trace: "com.google.devtools.build.lib.vfs.Path.<init>(Path.java:78)"
    stack_trace: "com.google.devtools.build.lib.vfs.Path.create(Path.java:73)"
    stack_trace: "com.google.devtools.build.lib.vfs.Path.create(Path.java:68)"
    stack_trace: "com.google.devtools.build.lib.vfs.FileSystem.getPath(FileSystem.java:72)"
    stack_trace: "com.google.devtools.build.lib.bazel.repository.downloader.UrlRewriter.newCredentialsFromNetrc(UrlRewriter.java:304)"
    stack_trace: "com.google.devtools.build.lib.bazel.BazelRepositoryModule.beforeCommand(BazelRepositoryModule.java:341)"
    stack_trace: "com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:379)"
    stack_trace: "com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:233)"
    stack_trace: "com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:550)"
    stack_trace: "com.google.devtools.build.lib.server.GrpcServerImpl.lambda$run$1(GrpcServerImpl.java:614)"
    stack_trace: "io.grpc.Context$1.run(Context.java:566)"
    stack_trace: "java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)"
    stack_trace: "java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)"
    stack_trace: "java.base/java.lang.Thread.run(Unknown Source)"
  }
}