Can you know what mutants were killed for each test case?

[islamelgendy]

Hi,

I need to find out if there is a way to link between the individual test cases and the killed mutants. After doing the mutation analysis, I can find out which mutants were killed/lived from the "kill.csv" file and I can know what were the actual mutants in the code from the "mutants.log" file. However, is there a way to find out for each test case, what mutants were killed? for example, say that for test001(), what are the mutants killed running this test case only?

[ekincanufuktepe]

Hi @islamelgendy

I think I can help you out with this one.

Under the defects4J directory, you can configure your defects4j.build.xml by adding the options and following the steps below:

• exportKillMap="true"
• sort="sort_methods" (this is the key part, by default the mutant coverage given is based on the test class not test method. You will notice that sort is set to original)
• You can also add the exportCovMap="true" if you want.
• Then, let's say you have you exported the fixed version of defects4j checkout -p Lang -v 1f -w /tmp/lang_1_fixed, and you change your directory to the exported project
• Simply run defect4j mutation
• Your testMap.csv file will contain your test method with their IDs. The mutant.log file will contain your mutants with their IDs. Finally, once you check your killMap.csv file will show you which test method kills which mutant based on their IDs.

Hopefully, I didn't miss any steps, and this would help you out!

Cheers!

[islamelgendy]

Hi @ekincanufuktepe

Thanks a lot for your reply. I'm a bit confused about certain things. I opened the defects4j.build.xml, and I tried to find the related section. I think it is the one titled 'Run mutation analysis'. The values are different from the ones you said. Here is the part where I believe you are speaking about:

<!-- Sort test classes by run time, unless a single test method is run -->
        <condition property="major.sort" value="original" else="sort_classes">
            <isset property="test.entry.method"/>
        </condition>

        <echo message="Running mutation analysis ..."/>
        <junit
            printsummary="false"
            showoutput="false"
            fork="no"

            mutationAnalysis="true"
            timeoutFactor="16"
            sort="${major.sort}"
            haltonfailure="true"
            killDetailsFile="${major.kill.log}"
            excludeFile="${major.exclude}"
            >

            <!--
            Setting haltonfailure to false leads to the exclusion of
            failing tests -> usually all tests should pass!
            -->

            <classpath>
                <pathelement location="${d4j.dir.classes.mutated}" />
                <pathelement path="${d4j.dir.classes.testgen}" />
                <path refid="d4j.test.classpath"/>
                <!-- Add dependencies to runtime libraries of test generation tools -->
                <path refid="d4j.lib.testgen.rt"/>
            </classpath>

            <test name="${test.entry.class}" methods="${test.entry.method}" if="test.entry.class" />
            <batchtest unless="test.entry.class" fork="no">
                <fileset refid="major.tests"/>
            </batchtest>
        </junit>

I see that inside the junit tag, the sort property is like this sort="${major.sort}". There is another property at the start in the condition tag value="original". Can you tell me which of these should be changed? Also, I'm guessing the exportKillMap="true" and exportCovMap="true" will be added to the properties of the junit tag, right?

[ekincanufuktepe]

Hi @islamelgendy,

Yes, that is the section where you have to do the necessary modifications. By default sort is linked to ${major.sort}, which is set to the value original. You can decide to keep that if you don't want to modify those settings. But, then you want to remove the sort="${major.sort}" line and replace it with sort="sort_methods". I mean that is how I preferred to do it.

I am sharing a snapshot from one of my examples just in case it helps.

Best,

Ekin

[islamelgendy]

Hi @ekincanufuktepe

I've made the changes to the xml file. However, the mutation now fails after setting the sort="sort_methods", while it was running successfully before.

Here is a screenshot of the error:

This is a screenshot of part of my defects4j.build.xml

Any thoughts?

[ekincanufuktepe]

Hi @islamelgendy,

The XML file looks good to me. I have a couple of guesses, but not sure what exactly the problem is. I need more information.

• Your Java version may not be compatible with Major. Can you share your Java version? Your Java version should be 8 as they mentioned in their documentation.
• I think you are running your generated tests by Randoop, again I am not sure if the issue is because of the generated tests. Can you also try using the developers' tests instead of the generated tests by Randoop?
• Make sure you are generating the mutants over the fixed versions and not the buggy versions.

Let me know what happens after trying the above.

Ekin