[BUG] Test is marked as successful even though it was not executed

[mardukbp]

I have a JUnit test for which the Maven goal testCompile (belonging to the maven-compiler-plugin) fails. Nevertheless, when the test is executed using the green play button in the gutter, the green triangle becomes a green tick. Even though the Debugging Console shows that the class was not found.

In addition to marking the test as failed it would be very useful to provide the output of the failed Maven execution in the Debugging Console. That the class was not found is a consequence of the compilation failure, and this stacktrace does not help to fix the problem. The output of Maven does.

[jdneo]

Is it possible to provide a sample project for this issue?

[mardukbp]

I learned that when a test is not executed the last test result is shown. After deleting %APPDATA%\Code\User\workspaceStorage\workspace-UUID\testResults I got the green triangle again. It would be good to have a command to clear this cache (Java: Clean Java Language Server Workspace does not do it). But in any case, if a test cannot be executed because the class does not exist I would expect that the test is marked as failed (though it would be better to have a Blocked status).

I added testCompile as prelaunchTask in order for the compilation error to be listed under Problems, thus solving my second point. But now I get a permanent toaster that reads: "Run Tests: Resolving launch configuration...". And every time I run the test another toaster appears above the previous one.

Here is the project: https://github.com/mardukbp/error-prone-demo

[jdneo]

It would be good to have a command to clear this cache

You can use Test: Clear All Results

My understanding is that you hope the tests should be marked as failed/error when the prelaunch task fails? Is that correct?

[mardukbp]

In my view a prelaunchTask is an explicit precondition and the existence of SomeTest.class is an implicit precondition. Whenever a precondition is not satisfied the test cannot be executed (in my project both preconditions fail). So strictly speaking the test itself doesn't fail, but a past execution result doesn't represent the current state of affairs.

The failure of the prelaunchTask is reported as an error in Problems and the runtime failure due to the missing class is reported in the Debugging Console. This information should be associated with the test execution and AFAIK marking the test as failed is a way to accomplish that.

[jdneo]

Ok, I got your point.

We need some kind of UI hint to let the users know tests are not executed due to some reasons (compile error, etc...). Currently displaying a pass status in the previous session is misleading.

[mardukbp]

And what about the toasters that don't go away when the execution fails?

Am 28. Oktober 2022 08:57:11 MESZ schrieb Sheng Chen @.***>:

Ok, I got your point.

We need some kind of UI hint to let the users know tests are not executed due to some reasons (compile error, etc...). Currently displaying a pass status in the previous session is misleading.

-- Reply to this email directly or view it on GitHub: https://github.com/microsoft/vscode-java-test/issues/1499#issuecomment-1294545314 You are receiving this because you authored the thread.

Message ID: @.***>

[github-actions[bot]]

Hi @mardukbp, I'm an AI Support assistant here to help with your issue. While the team reviews your request, I wanted to provide some possible tips and documentation that might help you in the meantime.

It looks like the Java Test Runner currently doesn’t treat a Maven compilation failure as a test-run error, nor surface the Maven logs in the Debug Console. You can work around this by explicitly compiling before running the test:

• Enable the built-in “compile before run” feature:

// settings.json
"java.test.compileBeforeRun": true

This makes the Test Runner invoke javac on your sources, but it won’t pick up Maven-specific compiler errors.

• Define a pre-launch task in your launch configuration to run mvn test-compile:

// .vscode/tasks.json
{
  "version": "2.0.0",
  "tasks": [
    {
      "label": "maven-test-compile",
      "type": "shell",
      "command": "mvn",
      "args": ["test-compile"],
      "problemMatcher": "$maven"
    }
  ]
}

// .vscode/launch.json
{
  "configurations": [
    {
      "type": "java",
      "request": "launch",
      "name": "Debug JUnit Test",
      "mainClass": "${file}",
      "preLaunchTask": "maven-test-compile"
    }
  ]
}

Now, if the Maven compile step fails, you’ll see its full output and the test won’t be marked green.

Since there’s no existing issue that tracks this exact behavior, please consider filing a feature request in the Java Test Runner repo to:

• Treat a failed Maven compile as a test failure
• Pipe the full Maven logs into the Debug Console

Other references with low confidence

• Overriden class in test package causes “type is already defined” error – describes how compile errors in test code stop the debugger from starting (https://github.com/redhat-developer/vscode-java/issues/1432)

The team will respond to your issue shortly. I hope these suggestions are helpful in the meantime. If this comment helped you, please give it a 👍. If the suggestion was not helpful or incorrect, please give it a 👎. Your feedback helps us improve!