[MCOMPILER-372] Unable to compile modularized test code depending on test-jar

[jira-importer]

Jeronimo Backes opened MCOMPILER-372 and commented

I'm refactoring univocity-parsers into multiple projects with modules (attached a zip with everything I got)

However I'm unable to reliably build the attached project with its unit tests. Unpack then cd into "univocity-all-parent", then run "mvn clean install".

 

All projects generate test-jars, but for some reason class com.univocity.parsers.core.routine.ResultSetWritingRoutine inside "univocity-parsers-core/src/test/java/" is not found when compiling the tests of projects "univocity-csv", "univocity-tsv" and "univocity-fixedwidth".

 

Interestingly, the issue doesn't seem to be consistent as I got "univocity-csv" and "univocity-tsv" building without errors a few times. "univocity-fixedwidth" however failed consistently every single time.

Some of the errors I get are:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:testCompile (default-testCompile) on project univocity-csv: Compilation failure: Compilation failure: [ERROR] /home/jbax/dev/repository/univocity-all/univocity-oss/univocity-csv/src/test/java/com/univocity/csv/CsvWriterTest.java:[27,49] cannot find symbol [ERROR]   symbol:   class ResultSetWritingRoutine [ERROR]   location: package com.univocity.parsers.core.routine [ERROR] /home/jbax/dev/repository/univocity-all/univocity-oss/univocity-csv/src/test/java/com/univocity/csv/CsvWriterTest.java:[681,17] cannot find symbol [ERROR]   symbol:   class ResultSetWritingRoutine [ERROR]   location: class com.univocity.csv.CsvWriterTest [ERROR] /home/jbax/dev/repository/univocity-all/univocity-oss/univocity-csv/src/test/java/com/univocity/csv/CsvWriterTest.java:[681,52] cannot find symbol [ERROR]   symbol:   class ResultSetWritingRoutine [ERROR]   location: class com.univocity.csv.CsvWriterTest [ERROR] /home/jbax/dev/repository/univocity-all/univocity-oss/univocity-csv/src/test/java/com/univocity/csv/CsvWriterTest.java:[691,17] cannot find symbol [ERROR]   symbol:   class ResultSetTest [ERROR]   location: class com.univocity.csv.CsvWriterTest [ERROR] /home/jbax/dev/repository/univocity-all/univocity-oss/univocity-csv/src/test/java/com/univocity/csv/CsvWriterTest.java:[691,45] cannot find symbol [ERROR]   symbol:   class ResultSetTest [ERROR]   location: class com.univocity.csv.CsvWriterTest [ERROR] /home/jbax/dev/repository/univocity-all/univocity-oss/univocity-csv/src/test/java/com/univocity/csv/routine/CsvRoutinesTest.java:[123,17] cannot find symbol [ERROR]   symbol:   class ResultSetWritingRoutine [ERROR]   location: class com.univocity.csv.routine.CsvRoutinesTest [ERROR] /home/jbax/dev/repository/univocity-all/univocity-oss/univocity-csv/src/test/java/com/univocity/csv/routine/CsvRoutinesTest.java:[123,52] cannot find symbol [ERROR]   symbol:   class ResultSetWritingRoutine [ERROR]   location: class com.univocity.csv.routine.CsvRoutinesTest [ERROR] /home/jbax/dev/repository/univocity-all/univocity-oss/univocity-csv/src/test/java/com/univocity/csv/routine/CsvRoutinesTest.java:[124,68] package ResultSetWritingRoutine does not exist

 

I was also getting errors saying that the "Example" class was not found, or that the "printAndValidate" method was not found (that one comes from the univocity-output-tester dependency)..

 

There's something very weird going on and it's not consistently reproducible. If you for example change the code in the failing tests use "import static com.univocity.parsers.core.routine.ResultSetWritingRoutine.* " you may get a different set of errors. It's pretty intractable.

I hope this provides enough information, let me know if you need anything else.

---

Affects: 3.8.0

Attachments:

• maven-compiler-plugin.MCOMPILER-372.patch (1.68 kB)
• univocity-all.zip (912.77 kB)

Issue Links:

• MCOMPILER-429 CLONE - Unable to compile modularized test code depending on test-jar ("is cloned by")
• MCOMPILER-429 CLONE - Unable to compile modularized test code depending on test-jar ("is duplicated by")

Remote Links:

• GitHub Pull Request #27

0 votes, 5 watchers

[jira-importer]

François Loison commented

I may have found a patch.

Plugin compileTest goal is looking for module-info class in non-test output folder.

As a consequence, dependentProject-tests.jar is not added in test compilation classpath.

I modified plugin to seach in test output folder.

module-info class is not found, dependentProject-tests.jar is added in test compilation classpath and test compilation succeeds.

Patch attached : [^maven-compiler-plugin.MCOMPILER-372.patch]

[jira-importer]

Robert Scholte commented

Can you create a pullrequest at https://github.com/apache/maven-compiler-plugin ? It will also ask you to provide an integration test to confirm the fix and to prevent regression in the future. Advice is to pick another IT from src/it as base.

[jira-importer]

François Loison commented

I wrote the src/it/MCOMPILER-372_modularized_testjar integration test case that reflects following use case:

Module 2 project using a modularized Module 1 project while Module 1 test jar is not modularized.

Unfortunately, my first fix breaks other integration projects.

I found out what I have to do : for each modularized dependent project, add test path (if exists) to --patch-module option.

 

Example:

--patch-module module1_372=..\module1\target\test-classes etc...

 

I'm still working on it.

 

[jira-importer]

François Loison commented

Fix found and tested OK with 'verify'. I'm trying to create a pull request.

[jira-importer]

François Loison commented

Here is pull request:

https://github.com/apache/maven-compiler-plugin/pull/27

[jira-importer]

Robert Scholte commented

The IT does indeed help to reproduce the issue. I've had a look at the output, especially at the --patch-module result, and it looks like it is not correct:

--patch-module prj2_372=
E:\java-workspace\apache-maven-plugins\plugins\maven-compiler-plugin\target\it\MCOMPILER-372_modularized_testjar\prj2\target\classes;
E:\java-workspace\apache-maven-plugins\plugins\maven-compiler-plugin\target\it\MCOMPILER-372_modularized_testjar\prj2\src\test\java;
E:\java-workspace\apache-maven-plugins\plugins\maven-compiler-plugin\target\it\MCOMPILER-372_modularized_testjar\prj2\target\generated-test-sources\test-annotations;

--patch-module prj1_372=
E:\java-workspace\apache-maven-plugins\plugins\maven-compiler-plugin\target\it\MCOMPILER-372_modularized_testjar\prj2\target\test-classes;
E:\java-workspace\apache-maven-plugins\plugins\maven-compiler-plugin\target\it\MCOMPILER-372_modularized_testjar\prj1\target\classes;
E:\java-workspace\apache-maven-plugins\plugins\maven-compiler-plugin\target\it\MCOMPILER-372_modularized_testjar\prj1\target\test-classes;
E:\java-workspace\apache-maven-plugins\plugins\maven-compiler-plugin\target\local-repo\junit\junit\3.8.2\junit-3.8.2.jar;

I would expect extra entries, not disappearing entries. And junit-3.8.2 shouldn't be patched right?

[jira-importer]

François Loison commented

I agree. I tested on a project structure of mine and patch broke test compilation for reasons equivallent to what you said.

I'll work on this again.

[jira-importer]

François Loison commented

Previous answer was partially correct.

I had a test-compile failure for a project containing a JAXB XSD to java generation. Old code was trying to patch unmodularized dependencies.

I added it 'prj0' project to test it.

With new fix in pull request, prj0 passes. Also, all it tests pass. ( https://github.com/apache/maven-compiler-plugin/pull/27 )

New code patches only dependencies with MODULEDESCRIPTOR.

I copy/pasted some methods from CompilerMojo. It may be shared if needed.

You told that junit-3.8.2 should not be patched.

With fix, there is no --patch-module junit=... but there is still --patch-module prj1_372=...;junit-3.8.2.jar

This is required to compile OK. For some reason, adding junit-3.8.2.jar to class path is not sufficient.

Here is compiler patch module arguments, paths are relative from prj2 for readability:

--patch-module prj2_372=
target\classes;
src\test\java;
target\generated-test-sources\test-annotations;

--patch-module prj1_372=
target\test-classes;
..\prj1\target\classes;
..\prj1\target\test-classes;
%LOCAL_REPO%\junit\junit\3.8.2\junit-3.8.2.jar;

For information, compiler path module argument without pull request is:

--patch-module prj2_372=
target\classes;
src\test\java;
target\generated-test-sources\test-annotations;

[jira-importer]

Robert Scholte commented

What's the exception you get without junit? And I noticed code for multirelease in test. That's unrelated to this issue and I wonder if we should support it for tests.

[jira-importer]

François Loison commented

The exception is this one:

javac -d target\test-classes ^ -classpath target\test-classes;..\prj1\target\test-classes;%LOCAL_REPO%\junit\junit\3.8.2\junit-3.8.2.jar; ^ --module-path target\classes;..\prj1\target\classes; ^ -sourcepath src\test\java;target\generated-test-sources\test-annotations; ^ -s target\generated-test-sources\test-annotations ^ -g -nowarn --release 11 -encoding UTF-8 ^ --patch-module prj2_372=target\classes;src\test\java;target\generated-test-sources\test-annotations; ^

( notice that junit.framework.TestCase is removed from following patch module ) --patch-module prj1_372=target\test-classes;..\prj1\target\classes;..\prj1\target\test-classes; ^ --add-reads prj2_372=ALL-UNNAMED ^ src\test\java\prj2\Prj2Test.java

src\test\java\prj2\Prj2Test.java:24: error: cannot access TestCase public class Prj2Test extends Prj1Test ^ class file for junit.framework.TestCase not found 1 error

OK to remove the multi-module code which is not relevant for test compile.

Unfortunately, I found another issue when test code uses 2 modules. I'm working on it.

[jira-importer]

Robert Scholte commented

I think the problem with junit is easy to fix: all patched modules should also have --add-reads for ALL-UNNAMED

[jira-importer]

François Loison commented

Thanks for the tip.

javac -d target\test-classes ^ -classpath target\test-classes;..\prj1\target\test-classes;%LOCAL_REPO%\junit\junit\3.8.2\junit-3.8.2.jar; ^ --module-path target\classes;..\prj1\target\classes; ^ -sourcepath src\test\java;target\generated-test-sources\test-annotations; ^ -s target\generated-test-sources\test-annotations ^ -g -nowarn --release 11 -encoding UTF-8 ^ --patch-module prj2_372=target\classes;src\test\java;target\generated-test-sources\test-annotations; ^ ( notice that junit.framework.TestCase is removed from following patch module ) --patch-module prj1_372=target\test-classes;..\prj1\target\classes;..\prj1\target\test-classes; ^ --add-reads prj2_372=ALL-UNNAMED ^ --add-reads prj1_372=ALL-UNNAMED ^ src\test\java\prj2\Prj2Test.java

Compile OK!

 

[jira-importer]

François Loison commented

Happy new year!

I commited a new version in pull request.

I build a list of test artifacts for each module.

Then it adds a --patch-module instruction for each module containing this list.

--add-reads <module>=ALL-UNNAMED is also added to access to unammed class (example junit3).

I added a prj3 test projet to test multi-modules inheritance.

New code passes all integration tests and pass with a complex multi-modules project of mine.

 

[jira-importer]

Robert Scholte commented

Just had another look at this issue. The fix as suggested is not the right approach: test-jars don't have to come from the reactor, they can also come from a remote repository, That shouldn't matter.

What I think what could work is to verify is in case of the same groupId + artifactId, these should be patched.