Wrong results mutating

[ivanvillarfreire]

Good afternoon.

I open this issue due to a very rare situation I'm facing on. I will try to explain it with simple examples:

I have this function:

    func divisionResultMajorThanOne(number1: Int, number2: Int) -> Bool {
        var isMajorThanOne = false
        let result = number1 / number2
        
        if (result > 1) {
            isMajorThanOne = true
        }

        return isMajorThanOne
    }

As you see, is a pretty simple function which is tested with this:

    func testDivisionIsMajorThanOne() {
        let result = basic?.divisionResultMajorThanOne(number1: 4, number2: 2)
        XCTAssertEqual(result, true, "Division Major Than One failed")
    }

So far, so good, the test works:

Now, I can do a "manual mutation" myself, let's transform the function to this one and re-execute the tests:

    func divisionResultMajorThanOne(number1: Int, number2: Int) -> Bool {
        var isMajorThanOne = false
        let result = number1 / number2
        
        if (result < 1) {
            isMajorThanOne = true
        }

        return isMajorThanOne
    }

Notice the change from:

 if (result > 1) {

To:

 if (result < 1) {

After reexecute the test it fails:

Which is obviously correct.

At this point, mutation-speaking, we could say that this mutant is killed when I simulate a "manual mutation" let's say.

Now, if I put back the original code and execute muter with:

muter run --files-to-mutate Utils/Basic.swift

The mutation suite ends ok, but:

Here's your test report:

--------------------------
Applied Mutation Operators
--------------------------

These are all of the ways that Muter introduced changes into your code.

In total, Muter introduced 1 mutants in 1 files.

File             Applied Mutation Operator       Mutation Test Result
----             -------------------------       --------------------
Basic.swift:21   RelationalOperatorReplacement   mutant survived


--------------------
Mutation Test Scores
--------------------

These are the mutation scores for your test suite, as well as the files that had mutants introduced into them.

Mutation scores ignore build errors.

Of the 1 mutants introduced into your code, your test suite killed 0.
Mutation Score of Test Suite: 0%
Muter could not gather coverage data from your project

File          # of Introduced Mutants   Mutation Score
----          -----------------------   --------------
Basic.swift   1                         0

That is the problem:

That is telling me that the mutant is not killed but that cannot be true according to the first test that I did manually, right? I can even see on the mutated code folder how indeed the change, as the report says, is the change from > to <:

    func divisionResultMajorTh/Users/ivanvillar/Documents/ProyectosInditex/mob-helloworldqaios/code_mutated/Utils/Basic.swiftanOne(number1: Int, number2: Int) -> Bool { if ProcessInfo.processInfo.environment["Basic_21_20_456"] != nil {
        var isMajorThanOne = false
        let result = number1 / number2
        
        if (result < 1) {
            isMajorThanOne = true
        }

        return isMajorThanOne
} else {
        var isMajorThanOne = false
        let result = number1 / number2
        
        if (result > 1) {
            isMajorThanOne = true
        }

        return isMajorThanOne
}

At this point I'm thinking on some kind of problem with the schemata way of working.

Other data:

executable: /opt/homebrew/bin/fastlane
arguments:
- scan
- --workspace
- HelloWorldE2E.xcworkspace
- --scheme 
- HelloWorldE2E_QA

MacBook-Pro-de-Ivan:code ivanvillar$ fastlane -v
fastlane installation at path:
/opt/homebrew/Cellar/fastlane/2.220.0/libexec/gems/fastlane-2.220.0/bin/fastlane
-----------------------------
[✔] 🚀 
fastlane 2.220.0

MacBook-Pro-de-Ivan:code ivanvillar$ swift --version
swift-driver version: 1.75.2 Apple Swift version 5.8 (swiftlang-5.8.0.124.2 clang-1403.0.22.11.100)
Target: arm64-apple-macosx13.0

[ZevEisenberg]

Hey @ivanvillarfreire - I don't have time to look at this right now, but I just wanted to pop in and thank you for filing a really nice bug report ❤️

[nandohendo]

Hi @ivanvillarfreire @ZevEisenberg , I might've encountered a similar issue. One thing I've tried to do is to print out ProcessInfo.processInfo.environment in my test suite, and I noticed that I can't find these entries: "IS_MUTER_RUNNING": "YES" and the usual "FileName_54_59_424": "YES", or in @ivanvillarfreire 's case would be something like "Basic_21_20_456": "YES" in the test log that Muter generates.

In this case, I would say that Muter has successfully inserted the mutant into the copy of the codebase, but it can't seem to run on the mutated part of the code.

Thank you!

[rakaramos]

hi @ivanvillarfreire thanks for the report! Could check the logs? I'm not able to reproduce it. Here is the project that I'm using

issue.zip

Command line invocation:
    /Applications/Xcode-15.4.0.app/Contents/Developer/usr/bin/xcodebuild test-without-building -destination platform=macOS,arch=x86_64 -xctestrun muter.xctestrun

User defaults from command line:
    IDEPackageSupportUseBuiltinSCM = YES

--- xcodebuild: WARNING: Using the first of multiple matching destinations:
{ platform:macOS, arch:x86_64, id:00006000-000A38D61E02401E, name:My Mac }
{ platform:macOS, arch:x86_64, variant:Mac Catalyst, id:00006000-000A38D61E02401E, name:My Mac }
Testing started
Test suite 'issueTests' started on 'My Mac - issue (18254)'
Test case 'issueTests.testDivisionIsMajorThanOne()' failed on 'My Mac - issue (18254)' (0.149 seconds)

Test session results, code coverage, and logs:
	/..../Test-issue-2024.06.25_11-52-53--0300.xcresult

Failing tests:
	issueTests.testDivisionIsMajorThanOne()

** TEST EXECUTE FAILED **

[nandohendo]

Hi @ivanvillarfreire , the issue doesn't happen again for me after I try building from source instead of using the homebrew version.

[ivanvillarfreire]

https://github.com/muter-mutation-testing/muter/issues/283#issuecomment-2189220722

Hi @rakaramos

I cannot execute your test locally, fails with:

MacBook-Pro-de-Ivan:issue ivanvillar$ xcodebuild test-without-building -destination platform=macOS,arch=x86_64 -xctestrun muter.xctestrun
2024-06-27 09:19:12.369 xcodebuild[58968:8562289] DVTCoreDeviceEnabledState: DVTCoreDeviceEnabledState_Disabled set via user default (DVTEnableCoreDevice=disabled)
Command line invocation:
    /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild test-without-building -destination platform=macOS,arch=x86_64 -xctestrun muter.xctestrun

User defaults from command line:
    IDEPackageSupportUseBuiltinSCM = YES

--- xcodebuild: WARNING: Using the first of multiple matching destinations:
{ platform:macOS, arch:x86_64, id:00006021-001808323AF0C01E }
{ platform:macOS, arch:x86_64, variant:Mac Catalyst, id:00006021-001808323AF0C01E }
2024-06-27 09:19:12.547 xcodebuild[58968:8562289] Writing error result bundle to /var/folders/z1/b_6s03sj3155q3dds_yvb9ym0000gn/T/ResultBundle_2024-27-06_09-19-0012.xcresult
xcodebuild: error: Failed to build workspace temporary with scheme Transient Testing.: Cannot load xctestrun file (/Users/ivanvillar/Downloads/issue/muter.xctestrun): The file “muter.xctestrun” couldn’t be opened because there is no such file.

[rakaramos]

@ivanvillarfreire that is weird, are you running it manually or using muter?

[ivanvillarfreire]

I think it was my bad and I launched the wrong command.

Could you please provide the builded Muter binary? I want to test if it's something related with how I build on M2.

[rakaramos]

sure, here you go muter.zip