defects4j icon indicating copy to clipboard operation
defects4j copied to clipboard

Replace the regex expression to remove all string/character and comments of failing test methods

Open jose opened this issue 6 years ago • 0 comments

The framework/util/rm_broken_tests.pl script parses the output of a test suite compilation/execution and fixes failing test methods by replacing each broken test method with a dummy test method in the source file of the corresponding test class. For instance, supposing the following test suite with a failing test method (test1):

public class TestFoo {
  @Test
  public void test0() {
    assertTrue(true);
  }

  @Test
  public void test1() {
    String str = new String();
    str = 123456789;
  }
}

is then transformed to:

public class TestFoo {
  @Test
  public void test0() {
    assertTrue(true);
  }

  @Test
  public void test1() {}
// Defects4J: flaky method
//   @Test
//   public void test1() {
//     String str = new String();
//     str = 123456789;
//   }
}

Basically, the script parses the compilation/execution output and identifies the line where the failing test method starts/ends and it uses a regex expression to remove all string/character and comments of the failing test method as they may contain unbalanced delimiters or brackets (which may cause problems to find the last } of the test method). Although the script has been working fine in general, two issues (#96 and #184) have been reported related to that regex expression. Perhaps a more robust solution would be to build a parser that computes a line-number table for all methods and then use that table to comment out failing test methods.

jose avatar Nov 23 '18 02:11 jose