Debugging "Unknown Source" and "Unsupported evaluation in unmanaged classpath entry" in VSCode.

[PhilDakin]

Discussed in https://github.com/scalameta/metals/discussions/5801

^{Originally posted by PhilDakin October 30, 2023} When running test suites from spark-rapids, Metals is unable to resolve my source files with the following error:

If I evaluate local variables in the stack frame, it works fine. But if I try to access e.g. a test suite class definition, I see the following error:

2023.10.31 00:46:31 ERROR [error response][evaluate]: Cannot evaluate because of java.lang.Exception: Unsupported evaluation in unmanaged classpath entry: /home/ec2-user/my-spark-rapids/spark-rapids/tests/target/spark311/test-classes.

Per the Scala documentation, "Unmanaged files are manually created files that are outside of the control of the build." I am surprised to see /home/ec2-user/my-spark-rapids/spark-rapids/tests/target/spark311/test-classes listed as unmanaged, because it is definitely an artifact created when the module is built.

What would you recommend for continued debugging of this issue? Since spark-rapids is using mvn and not sbt, it isn't clear to me how the unmanaged/managed concept from the SBT docs applies here.

--

Promoting this to an issue because other users (@AndersSteenNilsen) in the Scalameta Discord reported the same problem. Anders claimed to have fixed the issue by adding the semanticdb-scalac dependency, but this was not sufficient for my case. @tdudzik indicates that the debugger is not stable.

Reproduction

• Get CUDA-enabled development environment.
• Checkout this spark-rapids branch.
• Setup bloop: ./build/buildall --generate-bloop --profile=311 + ln -s .bloop-spark311/ .bloop
• Install Metals, decline build import so that it uses our bloop install.
• Debugging any test suite will show "Unknown Source" error.

What I've Tried Already

• Adding semanticdb-scalac as a dependency.
• Enabling "metals.trace.server": "verbose" and inspecting the language server requests. Did not see anything indicating why the source is unknown.

What I Plan to Try Next

• See if issue persists in a more straightforward reproduction (e.g., running tests from the Metals repo itself).
• Building the Metals server locally and using print statements to determine exactly why the Metals server is unable to resolve the source.

[PhilDakin]

Able to see that the source is unknown in the debugger here: https://github.com/scalameta/metals/blob/684e67fcb9cbbf10e312eacdda92756d575e106c/metals/src/main/scala/scala/meta/internal/metals/debug/DebugProxy.scala#L207

Will continue working backwards

[PhilDakin]

Hm I noticed that only some of the project's targets get sources files generated in ~/.m2/repository. These targets see, to resolve properly.

Will try seeing why some targets don't get any sources files in the local repository as a next step.

[PhilDakin]

https://github.com/scalameta/metals/assets/15946757/9e0d1a36-8ed7-4915-b73e-401a9f56acd9

I have a similar setup to spark-rapids but lighter-weight available by:

1. git clone [email protected]:PhilDakin/scala-maven-example.git && cd scala-maven-example && git checkout test_branch
2. JAVA_HOME=<Java_11> ./buildall.sh

If you clone this, build it, install Metals, and use the .bloop config generated by buildall.sh (don't import project), you'll see "Unknown Source" when going to debug AnotherComponentTest.

The project structure of this example is similar to spark-rapids in that it is:

parent
-> lib, main + tests
-> lib_agg, just shaded JAR of lib
-> tests, tests only (no main), depends on lib_agg's shaded JAR

[gerashegalov]

Thanks for filing this issue @PhilDakin and creating a repro.

I would like to add a simplified repro for spark-rapids itself that sets aside complications with GPU setup and our non-trivial shaded artifacts.

We are going to utilize the fact that we can build spark-rapids on a regular CPU-only Linux host, and there are CPU-only tests in the sql-plugin submodule that will not require GPUs

1. clone spark-rapids and use branch-23.12 after this commit https://github.com/NVIDIA/spark-rapids/commit/525c73e4396ae71e82aa8cf4f3da48733a907fce

2. Build and bloopInstall just sql-plugin and sql-plugin-api submodules

JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 \
  mvn -pl sql-plugin -am clean package ch.epfl.scala:bloop-maven-plugin:bloopInstall \
  -DdownloadSources=true -Dbloop.configDirectory=$PWD/.bloop \
  -DskipTests -Dmaven.repo.local=$PWD/.bloop-m2

3. Start VSCode, open Metals tab, "Clean and restart build server",
4. Make sure "Run doctor" is all green
5. Open ArmSuite.scala which is CPU only
6. Make sure it's runnable without a debugger by clicking a play button or executing in the command palette Tests: Run Tests in Current File
7. Set a breakpoint inside Arm.scala e.g. on L38
8. Now execute Test: Debug Tests in Current File in ArmSuite.scala and observe the issue of "Unknown Source" for the Arm.scala class from the production source path

[PhilDakin]

This can be debugged further by using VSCode settings "metals.bloopVersion" and "metals.bloopJvmProperties" with a locally published Bloop version + "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005,quiet=y".

Bloop passes information to the scala-debug-adapter here, then the debug adapter initializes a SourceLookUpProvider here.

My understanding is that the SourceLookUpProvider gives a mapping from all fully-qualified class names (FQCN) to source files.

What's still unclear to me:

1 - In the original example, the CostBasedOptimizerSuite is provided in the sources field of the .bloop/X-test.json file, so it should be available in the SourceLookUpProvider. 2 - Are JARs for entries in the dependencies field of the .bloop/X.json file provided when constructing the SourceLookUpProvider?

Hitting my time box on this so will roll off to other things, but I'm pretty curious what the final issue is here. Perhaps there's something high-level obvious wrong in the bloop file that I'm not seeing just due to being relatively new to the Scala ecosystem.

[gerashegalov]

Another data point: there is some non-determinism in terms what projects are unlucky to hit "unknown source". Running the same example now flipped it: the product project can map to sources but the test project cannot.

[kasiaMarek]

I wasn't able to reproduce this using the small reproduction provided by @PhilDakin, so I was looking at example provided by @gerashegalov. In sql-plugin module root directory of the project (spark.rapids.source.basedir) is added as a resource (visible as a resource in bloop import in rapids-4-spark-sql_2.12.json). So the root directory is added to the classpath when debugging as an unmanaged entry (ch.epfl.scala.debugadapter.UnmanagedEntry) - one without sources. This seems to "overshadow" the correctly added modules (ch.epfl.scala.debugadapter.Module) with sources.

Update: Here the lookup for mapping fully qualified path name to classpath entry is created in scala-debug-adapter. Since here managed entries are first, due to how map is created the unmanned entries overshadow the managed ones. Changing the order of entries fixes the issue.

[gerashegalov]

@kasiaMarek thank you for looking into this issue.

It sounds like bloop import ignores that we include only specific files from spark.rapids.source.basedir https://github.com/NVIDIA/spark-rapids/blob/d3dc49679004e56a515ad1d0e5ab27a113ae1b56/sql-plugin/pom.xml#L143C32-L146

We can probably work it around by avoiding declaring these files as resources.

[kasiaMarek]

@gerashegalov might be worth creating an issue in: https://github.com/scalacenter/bloop-maven-plugin

[adpi2]

Here the lookup for mapping fully qualified path name to classpath entry is created in scala-debug-adapter. Since here managed entries are first, due to how map is created the unmanned entries overshadow the managed ones. Changing the order of entries fixes the issue.

Thanks a lot @kasiaMarek for investigating this. In the JDK, if two entries contain the same class name, the URLClassloader loads the first one. So propably we should reverse the allLookUps here to have a similar behavior. And that would also coincidentally fix the current issue.

[adpi2]

Closing this issue after creating https://github.com/scalacenter/scala-debug-adapter/pull/742 and https://github.com/scalacenter/bloop-maven-plugin/issues/85