dspot icon indicating copy to clipboard operation
dspot copied to clipboard

Published 20 hours ago • verified publisher icon STAMP-project

PERF: Remove dependencies to Maven / Gradle

Open danglotb opened this issue 5 years ago • 13 comments

Today, we use Maven and Gradle to: 1) compiles the sources, 2) collects the classpath and 3) execute pitest.

This dependency is a serious source of bugs, misunderstanding and is time-consuming.

We could totally remove these dependencies.

  1. we ask kindly the user to pre-compile its projects or we give the possibility to use Maven / Gradle to do so.
  2. we ask kindly the user to provide the classpath thought command line or properties and let the possibility to use Maven / Gradle to do so.
  3. we use the API of pitest / descartes to execute pit, see my old PR https://github.com/STAMP-project/dspot/pull/188 and hcoles/pitest/issues/405

danglotb avatar Mar 05 '19 14:03 danglotb

Asking myself if the execution of pitest should be in a new JVM, and if so, implemented in the STAMP-project/testrunner.

danglotb avatar Mar 06 '19 08:03 danglotb

Working on this lately. I just merged STAMP-project/testrunner#67 which implement a way to execute PIT in the test runner.

The next steps are:

  • [ ] create common API to either use the new way or the old way to execute PIT.
  • [x] create an optional command-line option to specify a classpath / dependencies (since we do not want to necessarily use maven / gradle to do so).
  • [ ] replace all old call to the current API by the newly created in the first task.

danglotb avatar Apr 04 '19 13:04 danglotb

Follow the changes in #753

danglotb avatar Apr 09 '19 10:04 danglotb

Working in progress: @danglotb @monperrus,

Hi Benjamin, I was trying to integrate Dspot with Defects4J to generate new tests for the Defects4j bugs.

  1. The first problem is the defects4j is built on Ant and relying on a lot of external dependencies, which means we can not use Maven/Gradle to execute the project.
  2. My solution is to add new input parameters of using existing classpath string. If the classpath is parsed by the user, we stop compileAndBuildClasspath in the method of initializeBuilder in this class. I tried to run the existing test cases in this way, they work fine.
  3. Please let me know if you have other solutions and if you would like me to send a pull request on this issue. Many thanks!

SophieHYe avatar Jul 11 '19 14:07 SophieHYe

Hello @SophieHYe

In fact, DSpot relies on Maven or Gradle for some tasks: cleaning and compiling, retrieve the classpath, run pit, etc.

My solution is to add new input parameters of using existing classpath string. If the classpath is parsed by the user, we stop compileAndBuildClasspath in the method of initializeBuilder in this class. I tried to run the existing test cases in this way, they work fine.

For me, it is perfectly fine to do so. However, this solution has several drawbacks:

  1. DSpot uses maven to retrieve the full classpath but also clean and compile (only one time). So if we introduce the command-line option that allows users to specify a classpath, we must assume that the classes and test classes are correctly compiled (also all the stuff like resources, generated code, etc that a mvn clean test would do).
  2. You won't be able to execute DSpot with PitMutantScoreSelector (the test-criterion based on mutation score) because this selector used either maven or gradle.

A better solution would be to implement a AutomaticBuilder that for Ant, let's say a AntAutomaticBuilder, such as the MavenAutomaticBuilder and the GradleAutomaticBuilder that implement the uses of maven and gradle respectively.

I remember this issue #317. @danzone Any progress?

In the meantime, I can add this command-line as fast as possible, but you'll have to handle to drawbacks.

Thank you.

danglotb avatar Jul 12 '19 08:07 danglotb

I just opened #834. It should do the trick ;)

danglotb avatar Jul 12 '19 09:07 danglotb

@danglotb Thanks a lot for the information.

You won't be able to execute DSpot with PitMutantScoreSelector (the test-criterion based on mutation score) because this selector used either maven or gradle.

That's true, I noticed the problem too.

A better solution would be to implement a AutomaticBuilder that for Ant, let's say a AntAutomaticBuilder, such as the MavenAutomaticBuilder and the GradleAutomaticBuilder that implement the uses of maven and gradle respectively.

Totally agree with you, it would be nice to implement an Ant builder!

I just opened #834. It should do the trick ;)

It works fine and many thanks!

SophieHYe avatar Jul 13 '19 09:07 SophieHYe

@danglotb, #834 does not work for me. Taking @SophieHYe usage example on the Defects4J dataset

work_dir="/tmp/Chart-1b/"
$ rm -rf "$work_dir"

# Get Chart-1b
$ "$D4J_HOME/framework/bin/defects4j" checkout -p "Chart" -v "1b" -w "$work_dir"
Checking out 2266 to /tmp/Chart-1b......................................... OK
Init local repository...................................................... OK
Tag post-fix revision...................................................... OK
Apply patch................................................................ OK
Run post-checkout hook..................................................... OK
Excluding broken/flaky tests............................................... OK
Excluding broken/flaky tests............................................... OK
Initialize fixed program version........................................... OK
Apply patch................................................................ OK
Initialize buggy program version........................................... OK
Diff 2266:2264............................................................. OK
Apply patch................................................................ OK
Tag pre-fix revision....................................................... OK
Check out program version: Chart-1b........................................ OK

# Compile it
$ cd "$work_dir"
$ "$D4J_HOME/framework/bin/defects4j" compile
Running ant (compile)...................................................... OK
Running ant (compile.tests)................................................ OK

# Get project's classpath
$ cd "$work_dir"
$ project_cp=$("$D4J_HOME/framework/bin/defects4j" export -pcp.test)
Running ant (export.cp.test)............................................... OK
$ echo "$project_cp"
/tmp/Chart-1b/lib/servlet.jar:/home/jose/defects4j/framework/projects/lib/junit-4.11.jar:/tmp/Chart-1b/build:/tmp/Chart-1b/build-tests

# Check whether the project's classpath is complete/valid
$ ls -la /tmp/Chart-1b/lib/servlet.jar
-rw-r--r-- 1 [email protected] domain [email protected] 80054 Aug 16 22:54 /tmp/Chart-1b/lib/servlet.jar
$ ls -la /home/jose/defects4j/framework/projects/lib/junit-4.11.jar
-rw-r--r-- 1 [email protected] domain [email protected] 288666 Jul 12 12:10 /home/jose/defects4j/framework/projects/lib/junit-4.11.jar
$ find /tmp/Chart-1b/build -type f -name "*.class" | head
/tmp/Chart-1b/build/org/jfree/chart/RenderingSource.class
/tmp/Chart-1b/build/org/jfree/chart/ui/FontDisplayField.class
/tmp/Chart-1b/build/org/jfree/chart/ui/Library.class
/tmp/Chart-1b/build/org/jfree/chart/ui/StrokeSample.class
/tmp/Chart-1b/build/org/jfree/chart/ui/FontChooserPanel.class
/tmp/Chart-1b/build/org/jfree/chart/ui/BasicProjectInfo.class
/tmp/Chart-1b/build/org/jfree/chart/ui/ExtensionFileFilter.class
/tmp/Chart-1b/build/org/jfree/chart/ui/StrokeChooserPanel.class
/tmp/Chart-1b/build/org/jfree/chart/ui/LCBLayout.class
/tmp/Chart-1b/build/org/jfree/chart/ui/StrokeChooserPanel$1.class
$ find /tmp/Chart-1b/build-tests -type f -name "*.class" | head
/tmp/Chart-1b/build-tests/org/jfree/chart/labels/junit/IntervalCategoryItemLabelGeneratorTests.class
/tmp/Chart-1b/build-tests/org/jfree/chart/labels/junit/ItemLabelAnchorTests.class
/tmp/Chart-1b/build-tests/org/jfree/chart/labels/junit/StandardCategoryItemLabelGeneratorTests.class
/tmp/Chart-1b/build-tests/org/jfree/chart/labels/junit/IntervalCategoryToolTipGeneratorTests.class
/tmp/Chart-1b/build-tests/org/jfree/chart/labels/junit/MultipleXYSeriesLabelGeneratorTests.class
/tmp/Chart-1b/build-tests/org/jfree/chart/labels/junit/StandardXYSeriesLabelGeneratorTests.class
/tmp/Chart-1b/build-tests/org/jfree/chart/labels/junit/CustomXYItemLabelGeneratorTests.class
/tmp/Chart-1b/build-tests/org/jfree/chart/labels/junit/StandardCategoryToolTipGeneratorTests.class
/tmp/Chart-1b/build-tests/org/jfree/chart/labels/junit/SymbolicXYItemLabelGeneratorTests.class
/tmp/Chart-1b/build-tests/org/jfree/chart/labels/junit/HighLowItemLabelGeneratorTests.class

# Get source code directories
$ cd "$work_dir"
$ source_dir=$("$D4J_HOME/framework/bin/defects4j" export -pdir.src.classes)
Running ant (export.dir.src.classes)....................................... OK
$ tests_dir=$("$D4J_HOME/framework/bin/defects4j" export -pdir.src.tests)
Running ant (export.dir.src.tests)......................................... OK
$ echo "$source_dir"
source
$ find "$source_dir" -type f -name "*.java" | head
source/org/jfree/chart/Effect3D.java
source/org/jfree/chart/ui/PaintSample.java
source/org/jfree/chart/ui/StrokeChooserPanel.java
source/org/jfree/chart/ui/StrokeSample.java
source/org/jfree/chart/ui/ExtensionFileFilter.java
source/org/jfree/chart/ui/Contributor.java
source/org/jfree/chart/ui/LCBLayout.java
source/org/jfree/chart/ui/FontDisplayField.java
source/org/jfree/chart/ui/Library.java
source/org/jfree/chart/ui/FontChooserPanel.java
$ echo "$tests_dir"
tests
$ find "$tests_dir" -type f -name "*.java" | head
tests/org/jfree/chart/labels/junit/StandardXYToolTipGeneratorTests.java
tests/org/jfree/chart/labels/junit/StandardPieSectionLabelGeneratorTests.java
tests/org/jfree/chart/labels/junit/BoxAndWhiskerXYToolTipGeneratorTests.java
tests/org/jfree/chart/labels/junit/IntervalCategoryItemLabelGeneratorTests.java
tests/org/jfree/chart/labels/junit/MultipleXYSeriesLabelGeneratorTests.java
tests/org/jfree/chart/labels/junit/HighLowItemLabelGeneratorTests.java
tests/org/jfree/chart/labels/junit/StandardXYSeriesLabelGeneratorTests.java
tests/org/jfree/chart/labels/junit/BoxAndWhiskerToolTipGeneratorTests.java
tests/org/jfree/chart/labels/junit/StandardXYZToolTipGeneratorTests.java
tests/org/jfree/chart/labels/junit/StandardCategoryToolTipGeneratorTests.java

# Get classes directories
$ cd "$work_dir"
$ classes_dir=$("$D4J_HOME/framework/bin/defects4j" export -pdir.bin.classes)
Running ant (export.dir.bin.classes)....................................... OK
$ test_classes_dir=$("$D4J_HOME/framework/bin/defects4j" export -pdir.bin.tests)
Running ant (export.dir.bin.tests)......................................... OK
$ echo "$classes_dir"
build
$ find "$classes_dir" -type f -name "*.class" | head
build/org/jfree/chart/RenderingSource.class
build/org/jfree/chart/ui/FontDisplayField.class
build/org/jfree/chart/ui/Library.class
build/org/jfree/chart/ui/StrokeSample.class
build/org/jfree/chart/ui/FontChooserPanel.class
build/org/jfree/chart/ui/BasicProjectInfo.class
build/org/jfree/chart/ui/ExtensionFileFilter.class
build/org/jfree/chart/ui/StrokeChooserPanel.class
build/org/jfree/chart/ui/LCBLayout.class
build/org/jfree/chart/ui/StrokeChooserPanel$1.class
$ echo "$test_classes_dir"
build-tests
$ find "$test_classes_dir" -type f -name "*.class" | head
build-tests/org/jfree/chart/labels/junit/IntervalCategoryItemLabelGeneratorTests.class
build-tests/org/jfree/chart/labels/junit/ItemLabelAnchorTests.class
build-tests/org/jfree/chart/labels/junit/StandardCategoryItemLabelGeneratorTests.class
build-tests/org/jfree/chart/labels/junit/IntervalCategoryToolTipGeneratorTests.class
build-tests/org/jfree/chart/labels/junit/MultipleXYSeriesLabelGeneratorTests.class
build-tests/org/jfree/chart/labels/junit/StandardXYSeriesLabelGeneratorTests.class
build-tests/org/jfree/chart/labels/junit/CustomXYItemLabelGeneratorTests.class
build-tests/org/jfree/chart/labels/junit/StandardCategoryToolTipGeneratorTests.class
build-tests/org/jfree/chart/labels/junit/SymbolicXYItemLabelGeneratorTests.class
build-tests/org/jfree/chart/labels/junit/HighLowItemLabelGeneratorTests.class

# Run DSpot
$ cd "$work_dir"
$ java -jar dspot-3.2.0-jar-with-dependencies.jar \
    --absolute-path-to-project-root "$work_dir" \
    --full-classpath "$project_cp" \
    --test "org.jfree.chart.axis.junit.LogAxisTests" \
    --test-cases "testTranslateJava2DToValue" \
    --output-directory "$work_dir/dspot-output" \
    --random-seed "23" \
    --relative-path-to-classes "$classes_dir" \
    --relative-path-to-source-code "$source_dir" \
    --relative-path-to-test-classes "$test_classes_dir" \
    --relative-path-to-test-code "$tests_dir" \
    --verbose

results in the following error

log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException:  (No such file or directory)
        at java.base/java.io.FileOutputStream.open0(Native Method)
        at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298)
        at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:237)
        at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:158)
        at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
        at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207)
        at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
        at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
        at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
        at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
        at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
        at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
        at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
        at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
        at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
        at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
        at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
        at org.slf4j.impl.Log4jLoggerFactory.<init>(Log4jLoggerFactory.java:66)
        at org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:72)
        at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:45)
        at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
        at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
        at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
        at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
        at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
        at eu.stamp_project.dspot.common.configuration.check.Checker.<clinit>(Checker.java:22)
        at eu.stamp_project.Main.parse(Main.java:46)
        at eu.stamp_project.Main.main(Main.java:15)
Exception in thread "main" java.lang.RuntimeException: java.io.FileNotFoundException: /tmp/Chart-1b/pom.xml (No such file or directory)
        at eu.stamp_project.dspot.common.automaticbuilder.maven.DSpotPOMCreator._innerCreatePom(DSpotPOMCreator.java:353)
        at eu.stamp_project.dspot.common.automaticbuilder.maven.DSpotPOMCreator.createNewPom(DSpotPOMCreator.java:141)
        at eu.stamp_project.dspot.common.automaticbuilder.maven.MavenAutomaticBuilder.<init>(MavenAutomaticBuilder.java:58)
        at eu.stamp_project.dspot.common.configuration.options.AutomaticBuilderEnum$1.getAutomaticBuilder(AutomaticBuilderEnum.java:18)
        at eu.stamp_project.dspot.common.configuration.InitializeDSpot.init(InitializeDSpot.java:48)
        at eu.stamp_project.dspot.DSpot.<init>(DSpot.java:33)
        at eu.stamp_project.Main.main(Main.java:19)
Caused by: java.io.FileNotFoundException: /tmp/Chart-1b/pom.xml (No such file or directory)
        at java.base/java.io.FileInputStream.open0(Native Method)
        at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
        at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
        at java.base/java.io.FileInputStream.<init>(FileInputStream.java:112)
        at java.base/sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:86)
        at java.base/sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:184)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:652)
        at java.xml/com.sun.org.apache.xerces.internal.impl.XMLVersionDetector.determineDocVersion(XMLVersionDetector.java:150)
        at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:860)
        at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:824)
        at java.xml/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
        at java.xml/com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:246)
        at java.xml/com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)
        at java.xml/javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:178)
        at eu.stamp_project.dspot.common.automaticbuilder.maven.DSpotPOMCreator._innerCreatePom(DSpotPOMCreator.java:327)
        ... 6 more

My system:

$ uname
Linux

$ java -version
openjdk version "1.8.0_292"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_292-b10)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.292-b10, mixed mode)

$ echo "$JAVA_HOME"
/home/jose/bin/jdk-8

$ ant -version
Apache Ant(TM) version 1.10.12 compiled on October 13 2021

$ echo "$ANT_HOME"
/home/jose/bin/apache-ant

$ mvn --version
Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
Maven home: /home/jose/bin/apache-maven
Java version: 1.8.0_292, vendor: AdoptOpenJDK, runtime: /home/jose/bin/jdk-8/jre
Default locale: en, platform encoding: UTF-8
OS name: "linux", version: "5.4.0-113-generic", arch: "amd64", family: "unix"

$ echo "$MAVEN_HOME"
/home/jose/bin/apache-maven

# [Defects4J v2.0.0](https://github.com/rjust/defects4j/tree/v2.0.0)

Am I executing DSpot incorrectly? Or is this some sort of regression bug?

jose avatar Aug 16 '22 22:08 jose

Hello,

Your command line seems correct. DSpot is trying to instrument the pom.xml. However, in JFreeChart, it seems that the pom.xml is not named as expected by DSpot, i.e. maven-jfreechart-project.xml :

ls ${work_dir}
ant  ChangeLog  checkstyle  defects4j.build.properties  docfiles  experimental  lib  licence-LGPL.txt  maven-jfreechart-project.xml  NEWS  README.txt  source  swt  tests

ATM, the name pom.xml is a constant, but we could imagine having a command-line parameter to be able to configure this.

Idk if @SophieHYe encountered this problem.

WDYT?

danglotb avatar Aug 17 '22 09:08 danglotb

Thanks @danglotb.

Just to make sure I understand it correctly, was not the purpose of --full-classpath to override whatever builder is defined?

--full-classpath=<dependencies>
        Specify the classpath of the project. If this option is used, DSpot won't use an
        AutomaticBuilder (e.g. Maven) to clean, compile and get the classpath of the project.
        Please ensure that your project is in a good shape, i.e. clean and correctly compiled,
        sources and test sources.

For Defects4J, in particular, I don't want DSpot to use any Maven or Gradle builder independently whether the pom.xml file exists (or any other pom.xml file named differently). Mostly, because I'm already providing/running DSpot with all project's metadata (e.g., location of class files, location of java files, project's classpath, etc) and the project is already in good shape as suggested in the --full-classpath's documentation.

jose avatar Aug 17 '22 10:08 jose

Hi,

The fact is that DSpot uses an automatic builder for several purposes.

The option --full-classpath overrides only the cleaning, compilation, and gathering the project's classpath at the start.

But other tasks remain dependent on the automatic builder, such as running pit to obtain the mutation score.

The problem is that during the construction of the MavenAutomaticBuilder object, it tries to instrument the pom.xml.

I guess that @SophieHYe avoided this by:

  1. Use the GradleAutomaticBuilder: --automatic-builder=Gradle, which might be strange as we do not want to use it.
  2. Use another test selection than the PitMutantScoreSelector, which relies on the AutomaticBuilder: for example --test-selector=JacocoCoverageSelector

Both MavenAutomaticBuilder and PitMutantScoreSelector are the ones used by default.

I am sorry that it is not convenient, but the option --full-classpath was a workaround and was not thought as a fully functional setup for DSpot.

I hope this is clearer for you, if not, please do not hesitate.

Thank you! :smile:

danglotb avatar Aug 17 '22 12:08 danglotb

Many thanks @danglotb.

If DSpot relies on Maven/Grandle to, e.g., run pit, I fail to see how @SophieHYe was able to run DSpot on all Defects4J's bugs. Perhaps she managed to run it only on those that do have a pom.xml file, e.g., the Math or Lang ones...

jose avatar Aug 17 '22 17:08 jose

Perhaps she managed to run it only on those that do have a pom.xml file, e.g., the Math or Lang ones...

Yes, it might be it. Perhaps, @SophieHYe uses the JacocoCoverageSelector which does not rely on the automatic builder but on an external library, a.k.a. test-runner.

danglotb avatar Aug 18 '22 04:08 danglotb