bazel crash during cquery (and possible pipeline invocation )

[comicfans]

Description of the bug:

I have some bazel 7.3.1 invocation like

bazel cquery "rdeps(//...,${bazel_path},1)" --config=debug --include_aspects=false | sed 's/ \(.*\)//g' | uniq | grep -v "${bazel_path}"  | bazel test --config=debug 

it tries to run the testcase which built from some cpp. and I found it crash some times with following error

FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.NullPointerException: Cannot invoke "com.google.devtools.build.lib.query2.common.CqueryNode.getLookupKey()" because "target" is null
        at com.google.devtools.build.lib.query2.cquery.ConfiguredTargetQueryEnvironment.getConfiguredTargetKey(ConfiguredTargetQueryEnvironment.java:513)
        at com.google.devtools.build.lib.query2.cquery.ConfiguredTargetQueryEnvironment.getConfiguredTargetKey(ConfiguredTargetQueryEnvironment.java:78)
        at com.google.devtools.build.lib.query2.PostAnalysisQueryEnvironment.unwindReverseDependencyDelegationLayers(PostAnalysisQueryEnvironment.java:417)
        at com.google.devtools.build.lib.query2.PostAnalysisQueryEnvironment.unwindReverseDependencyDelegationLayersIfFound(PostAnalysisQueryEnvironment.java:404)
        at com.google.devtools.build.lib.query2.PostAnalysisQueryEnvironment.skipDelegatingAncestors(PostAnalysisQueryEnvironment.java:378)
        at com.google.devtools.build.lib.query2.PostAnalysisQueryEnvironment.getReverseDeps(PostAnalysisQueryEnvironment.java:325)
        at com.google.devtools.build.lib.query2.PostAnalysisQueryEnvironment.getReverseDeps(PostAnalysisQueryEnvironment.java:108)
        at com.google.devtools.build.lib.query2.engine.AllRdepsFunction.lambda$eval$0(AllRdepsFunction.java:99)
        at com.google.devtools.build.lib.query2.common.AbstractBlazeQueryEnvironment.lambda$eval$0(AbstractBlazeQueryEnvironment.java:268)
        at com.google.common.util.concurrent.CombinedFuture$CallableInterruptibleTask.runInterruptibly(CombinedFuture.java:196)
        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:75)
        at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
        at com.google.common.util.concurrent.CombinedFuture$CombinedFutureInterruptibleTask.execute(CombinedFuture.java:108)
        at com.google.common.util.concurrent.CombinedFuture.handleAllCompleted(CombinedFuture.java:65)
        at com.google.common.util.concurrent.AggregateFuture.processCompleted(AggregateFuture.java:301)
        at com.google.common.util.concurrent.AggregateFuture.decrementCountAndMaybeComplete(AggregateFuture.java:283)
        at com.google.common.util.concurrent.AggregateFuture.lambda$init$0(AggregateFuture.java:157)
        at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
        at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
        at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:760)
        at com.google.common.util.concurrent.FluentFuture$TrustedFuture.addListener(FluentFuture.java:114)
        at com.google.devtools.build.lib.query2.common.AbstractBlazeQueryEnvironment$QueryTaskFutureImpl.addListener(AbstractBlazeQueryEnvironment.java:678)
        at com.google.common.util.concurrent.AggregateFuture.init(AggregateFuture.java:140)
        at com.google.common.util.concurrent.CombinedFuture.<init>(CombinedFuture.java:55)
        at com.google.common.util.concurrent.Futures$FutureCombiner.call(Futures.java:744)
        at com.google.devtools.build.lib.query2.common.AbstractBlazeQueryEnvironment.whenAllSucceedCall(AbstractBlazeQueryEnvironment.java:358)
        at com.google.devtools.build.lib.query2.common.AbstractBlazeQueryEnvironment.whenSucceedsCall(AbstractBlazeQueryEnvironment.java:346)
        at com.google.devtools.build.lib.query2.common.AbstractBlazeQueryEnvironment.eval(AbstractBlazeQueryEnvironment.java:265)
        at com.google.devtools.build.lib.query2.engine.AllRdepsFunction.eval(AllRdepsFunction.java:84)
        at com.google.devtools.build.lib.query2.engine.RdepsFunction.lambda$evalWithBoundedDepth$1(RdepsFunction.java:126)
        at com.google.devtools.build.lib.query2.common.AbstractBlazeQueryEnvironment.transformAsync(AbstractBlazeQueryEnvironment.java:386)
        at com.google.devtools.build.lib.query2.engine.RdepsFunction.evalWithBoundedDepth(RdepsFunction.java:128)
        at com.google.devtools.build.lib.query2.engine.RdepsFunction.eval(RdepsFunction.java:74)
        at com.google.devtools.build.lib.query2.engine.FunctionExpression.eval(FunctionExpression.java:55)
        at com.google.devtools.build.lib.query2.common.AbstractBlazeQueryEnvironment.eval(AbstractBlazeQueryEnvironment.java:264)
        at com.google.devtools.build.lib.query2.common.AbstractBlazeQueryEnvironment.evalTopLevelInternal(AbstractBlazeQueryEnvironment.java:141)
        at com.google.devtools.build.lib.query2.common.AbstractBlazeQueryEnvironment.evaluateQueryInternal(AbstractBlazeQueryEnvironment.java:185)
        at com.google.devtools.build.lib.query2.PostAnalysisQueryEnvironment.evaluateQuery(PostAnalysisQueryEnvironment.java:176)
        at com.google.devtools.build.lib.buildtool.PostAnalysisQueryProcessor.doPostAnalysisQuery(PostAnalysisQueryProcessor.java:211)
        at com.google.devtools.build.lib.buildtool.PostAnalysisQueryProcessor.process(PostAnalysisQueryProcessor.java:94)
        at com.google.devtools.build.lib.buildtool.BuildTool.buildTargets(BuildTool.java:211)
        at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:510)
        at com.google.devtools.build.lib.buildtool.BuildTool.processRequest(BuildTool.java:478)
        at com.google.devtools.build.lib.runtime.commands.CqueryCommand.exec(CqueryCommand.java:185)
        at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.execExclusively(BlazeCommandDispatcher.java:664)
        at com.google.devtools.build.lib.runtime.BlazeCommandDispatcher.exec(BlazeCommandDispatcher.java:244)
        at com.google.devtools.build.lib.server.GrpcServerImpl.executeCommand(GrpcServerImpl.java:573)
        at com.google.devtools.build.lib.server.GrpcServerImpl.lambda$run$1(GrpcServerImpl.java:644)
        at io.grpc.Context$1.run(Context.java:566)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)

Which category does this issue belong to?

C++ Rules, CLI, Local Execution

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

not always reproducible, someting like

bazel_path="//src/mod1/tests:test1.cpp"
bazel cquery "rdeps(//...,${bazel_path},1)" --config=debug --include_aspects=false | sed 's/ \(.*\)//g' | uniq | grep -v "${bazel_path}"  | bazel test --config=debug 

Which operating system are you running Bazel on?

Ubuntu 22.04.5 LTS 64bit

What is the output of bazel info release?

release 7.3.1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

maybe https://github.com/bazelbuild/bazel/issues/23022

Any other information, logs, or outputs that you want to share?

No response

[C0mbatwombat]

This is crashing very regularly for us in CI.

[comicfans]

any guide on how to debug this internal java issue ? I tried to launch my bazel with --host_jvm_debug and use jvm debugger attach, and break on uncaught NullPointerException, I can see lots of skyframe thread but if this issue happen, my jvm debug session just disconnect and not trap in debugger