-Dtest=Xxxxxx is also evaluated on non test plugins and thus fails

[cdietrich]

when testing with the tycho 2.4 rc i see the following behaviour. with have this pom in a sample project

<project xmlns="http://maven.apache.org/POM/4.0.0"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<parent>
		<groupId>org.eclipse.xtext</groupId>
		<artifactId>org.eclipse.xtext.swtbot.parent</artifactId>
		<version>1.0.0-SNAPSHOT</version>
		<relativePath>..</relativePath>
	</parent>

	<artifactId>org.eclipse.xtext.swtbot.testing</artifactId>
	<packaging>eclipse-plugin</packaging>

</project>

with tycho 2.4 the build fails with

2021-07-19T18:03:49.8467802Z [INFO] No tests to run.
2021-07-19T18:03:49.8468597Z [INFO] ------------------------------------------------------------------------
2021-07-19T18:03:49.8469761Z [INFO] Reactor Summary for org.eclipse.xtext.swtbot.parent 1.0.0-SNAPSHOT:
2021-07-19T18:03:49.8470536Z [INFO] 
2021-07-19T18:03:49.8471346Z [INFO] org.eclipse.xtext.swtbot.parent .................... SUCCESS [  2.100 s]
2021-07-19T18:03:49.8472490Z [INFO] org.eclipse.xtext.integrationtests.target .......... SUCCESS [  0.434 s]
2021-07-19T18:03:49.8473642Z [INFO] org.eclipse.xtext.swtbot.testing ................... FAILURE [  5.509 s]
2021-07-19T18:03:49.8474630Z [INFO] org.eclipse.xtext.swtbot.tests ..................... SKIPPED
2021-07-19T18:03:49.8475643Z [INFO] ------------------------------------------------------------------------
2021-07-19T18:03:49.8476244Z [INFO] BUILD FAILURE
2021-07-19T18:03:49.8476998Z [INFO] ------------------------------------------------------------------------
2021-07-19T18:03:49.8477617Z [INFO] Total time:  02:32 min
2021-07-19T18:03:49.8478279Z [INFO] Finished at: 2021-07-19T18:03:49Z
2021-07-19T18:03:49.8479060Z [INFO] ------------------------------------------------------------------------
2021-07-19T18:03:49.8481008Z [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.22.2:test (default-test) on project org.eclipse.xtext.swtbot.testing: No tests were executed!  (Set -DfailIfNoTests=false to ignore this error.) -> [Help 1]

[cdietrich]

log with 2.3 https://pipelines.actions.githubusercontent.com/VGbhG7MqInqDfxY0bKv0qAuPeX3d22owq0UvtAsHMpYjQuu8bd/_apis/pipelines/1/runs/535/signedlogcontent/25?urlExpires=2021-07-19T18%3A39%3A27.3935564Z&urlSigningMethod=HMACV1&urlSignature=JQCi7A3MBQr3x%2FAFsIJAMp9FFUd%2Fe52ZKey5a8l%2F5LI%3D log with 2.4 https://pipelines.actions.githubusercontent.com/VGbhG7MqInqDfxY0bKv0qAuPeX3d22owq0UvtAsHMpYjQuu8bd/_apis/pipelines/1/runs/536/signedlogcontent/20?urlExpires=2021-07-19T18%3A24%3A50.0389550Z&urlSigningMethod=HMACV1&urlSignature=jUrJUoHgrSxVmejPDeJNHJ8RVPkk1g3ahMnzQJbnouo%3D

[cdietrich]

@laeubi @mickaelistria any idea?

[laeubi]

@cdietrich I don't think it is the name but probably the new feature in Tycho 2.4 that executes test source folders as tests, can you give a link to the plugin/repro where this is located? I'll then take a look at this.

[cdietrich]

https://github.com/itemis/xtext-reference-projects/tree/cd_testTycho240 start script is https://github.com/itemis/xtext-reference-projects/blob/0d61b48197ade51e06c8aba763af030e70133fe7/.github/workflows/build.yml#L154 dont see any test attribute here https://github.com/itemis/xtext-reference-projects/blob/cd_testTycho240/integrationtests/org.eclipse.xtext.swtbot.testing/.classpath

[laeubi]

You are right, it seems somewhere in the call there is a request to execute some specific test.

Due to a bug in the suferfire plugin adding -Dsurefire.failIfNoSpecifiedTests=false should solve that issue.

[cdietrich]

can confirm that the workaround seems to work

[laeubi]

@cdietrich any chance you can test if the issue goes away with SUREFIRE 3.0.0-M6?

[cdietrich]

how can i choose to pick it?

[laeubi]

Should be something like adding

<dependency>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>3.0.0-M6</version>
</dependency>

to your pom but you probably need to build a local copy of the plugin yourself as I can't find any info about a snapshot repro here.

[cdietrich]

will try later this week

[cdietrich]

@laeubi unfortunately problem is also there with the M6 snapshot (can be found in https://repository.apache.org/content/repositories/snapshots)

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M6-SNAPSHOT:test (default-test) on project org.eclipse.xtext.swtbot.testing: No tests matching pattern "XtendExamplesTest" were executed! (Set -Dsurefire.failIfNoSpecifiedTests=false to ignore this error.) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M6-SNAPSHOT:test (default-test) on project org.eclipse.xtext.swtbot.testing: No tests matching pattern "XtendExamplesTest" were executed! (Set -Dsurefire.failIfNoSpecifiedTests=false to ignore this error.)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: org.apache.maven.plugin.MojoFailureException: No tests matching pattern "XtendExamplesTest" were executed! (Set -Dsurefire.failIfNoSpecifiedTests=false to ignore this error.)
    at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute (AbstractSurefireMojo.java:941)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)

[laeubi]

At least the error message is now much more clear ... I'll try to get in touch with the Surefireteam once again if this is intentional.

[cdietrich]

but dont you use the 2.22 for a long time? also in 2.4 i assume something has changed with 2.5 so maybe now tycho applies the plugin to more places than before

[laeubi]

Before, tests where simply never executed for eclipse-plugin packaging types, this has changed with 2.4.x release

The logic works (or at least is expected to work) that way:

• if you eclipse-plugin defines any 'test' scoped source then the maven-surefire plugin kicks in at the 'test' stage if anything matches the usual test-patterns, if not, nothing happens
• if your eclipse-plugin defines any 'test' scoped sources and matches the Integrationpttern (*IT.java) it executes the tycho-surefire at the integration-test and the validation at the validate stage, if not nothing happens

All this matches the usual maven behavior in contrast to the traditional test-plugin packaging of tycho where all tests happen and verified at the 'integration-test' stage (what is, from maven POV simply wrong) and always requires to have dedicated test-plugin projects. This is especially a problem with tycho-pomless as we "guess" whether or not a bundle is a test-bundle by its naming.

What happens here is, that some where in your poms/build setup you define to run only tests matching the pattern XtendExamplesTest (what is (until now) only intended to be specified only on a single module as far as I have understood the code!) on a global level.

Because of that the surefireplugin currently throws an error as you specify to run a test matching XtendExamplesTest but not all modules contain such a test.

You will get the same error on a plain maven project (see reproducer here .

[cdietrich]

what is a " 'test' scoped source " i assume there is some magic in platform that matches testing as i initially have guessed?!?

[laeubi]

I have described the user-story with a screenshot a bit more detailed in the release notes. Let me know if anything is missing or unclear so we can adjust, or simply hit the edit button to suggest a change right now from within github.

[cdietrich]

yes but we dont do that https://github.com/itemis/xtext-reference-projects/blob/master/integrationtests/org.eclipse.xtext.swtbot.testing/.classpath will try to rename the plugin for counter test

[laeubi]

correct, that's why I wrote

if not, nothing happens

and that's exactly what you see with -Dsurefire.failIfNoSpecifiedTests=false, this property only prevents surefire from failing if

1. no test are there
2. but a pattern (here XtendExamplesTest) are specified somewhere

So you can also circumvent this by removing the specification of the XtendExamplesTest pattern, but I guess its there for a reason...

[cdietrich]

we specify the XtendExamplesTest intentionally cause the tests have a long runtime and we want to run them in isolation.

[laeubi]

Maybe profiles that specify modules to execute, -DskipTest or configuration of the mojos are better alternatives than.

Anyway surefire.failIfNoSpecifiedTests=false should give you at least pre-2.4 behavior unless maven-sure-fire issue is resolved.

[cdietrich]

name indeed does not play a role.

[laeubi]

Yep, you can call it as you like :-)

[laeubi]

@cdietrich is this still relevant or can we close it?