mull-project
How to mutate data defined in code under test with Mull?
Description
Question: Is there any functionality in Mull that can mutate data like defined in the C++ array:
static const std::array<int,12> daysInMonthArray = {31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31};
and will detect if the test suite can detect changes in this data?
Background:
We have a research project that is trying to use AI coding agents to automatically generate unit tests for legacy C++ code. We are researching providing feedback to the LLM/AI agent to help it produce better tests that lock down the behavior of the code (so we can safely modify and extend it with fast feedback as part of the Legacy Code Change Algorithm). Between LLVM source-based coverage tooling (providing line, branch, and even MC/DC coverage metrics) and Mull (providing mutation testing for most C++ operators), we feel we can provide the LLM/AI agent with enough feedback to give it a good shot at generating solid characterization tests that lock down the behavior of the code we are testing. At the very least, these tools provide a way to grade the quality of the generated test suite to determine if we have gaps in coverage or checking of the outputs (as part of our benchmarking framework and for eventual production usage).
However, one remaining gap is our plan is checking for data defined in the software we are testing.
Example:
For example, suppose I have the simple function returning the number of days in a month (considering leap years):
int getNumDaysInMonth(int monthID, bool isLeapYear) {
static const std::array<int,12> daysInMonthArray = {31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31};
int daysInMonth = daysInMonthArray[monthID];
if (monthID == 1 && isLeapYear) { // Feb in leap year!
daysInMonth += 1;
}
return daysInMonth;
}
The following Google Test unit tests provide 100%, line, branch, and MC/DC coverage (as determined by Clang source-based coverage tooling):
TEST(GetNumDaysInMonthTest, JanLeapYear) {EXPECT_EQ(getNumDaysInMonth(0, true), 31);}
TEST(GetNumDaysInMonthTest, FebLeapYear) {EXPECT_EQ(getNumDaysInMonth(1, true), 29);}
TEST(GetNumDaysInMonthTest, FebNonLeapYear) {EXPECT_EQ(getNumDaysInMonth(1, false), 28);}
In addition, setting:
mutators:
- cxx_all
in my mull..yml file and with Mull as of version:
$ mull-runner-19 --version
Mull: Practical mutation testing for C and C++
Home: https://github.com/mull-project/mull
Docs: https://mull.readthedocs.io
Support: https://mull.readthedocs.io/en/latest/Support.html
Version: 0.26.1
Commit: f24030d
Date: 12 Oct 2025
LLVM: 19.1.6
these three unit tests squash all of the mutations generated by Mull with cxx_all:
$ mull-runner-19 --coverage-info=coverage.profdata ./getNumDaysInMonth_UnitTests_full_cov_strong
[info] Using config /mounted_from_host/numerical_coverage_examples/BUILDS/clang-19-cov-mull/mull.yml
[warning] Could not find dynamic library: libstdc++.so.6
[warning] Could not find dynamic library: libm.so.6
[warning] Could not find dynamic library: libgcc_s.so.1
[warning] Could not find dynamic library: libc.so.6
[info] Warm up run (threads: 1)
[################################] 1/1. Finished in 105ms
[info] Filter mutants (threads: 1)
[################################] 1/1. Finished in 0ms
[info] Baseline run (threads: 1)
[################################] 1/1. Finished in 104ms
[info] Running mutants (threads: 3)
[################################] 3/3. Finished in 101ms
[info] All mutations have been killed
[info] Mutation score: 100%
[info] Total execution time: 310ms
But you can change any of the elements of the array daysInMonthArray other than daysInMonthArray[0] and daysInMonthArray[1] and these test will pass. For example, you can set the last 10 elements in daysInMonthArray to 0 with the change:
diff --git a/getNumDaysInMonth/getNumDaysInMonth.cpp b/getNumDaysInMonth/getNumDaysInMonth.cpp
index d26270d..28cceab 100644
--- a/getNumDaysInMonth/getNumDaysInMonth.cpp
+++ b/getNumDaysInMonth/getNumDaysInMonth.cpp
@@ -8,7 +8,7 @@ int getNumDaysInMonth(int monthID, bool isLeapYear) {
//std::array<int,12> daysInMonthArray = {31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31};
//const std::array<int,12> daysInMonthArray = {31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31};
//static std::array<int,12> daysInMonthArray = {31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31};
- static const std::array<int,12> daysInMonthArray = {31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31};
+ static const std::array<int,12> daysInMonthArray = {31, 28, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 };
int daysInMonth = daysInMonthArray[monthID];
if (monthID == 1 && isLeapYear) { // Feb in leap year!
daysInMonth += 1;
those three tests that squash every Mull mutation still pass.
To cover all of the data defined in that future, you need 10 additional unit tests, one for each of the remaining 10 months:
...
TEST(GetNumDaysInMonthTest, MarNonLeapYear) {EXPECT_EQ(getNumDaysInMonth(2, false), 31);}
TEST(GetNumDaysInMonthTest, AprNonLeapYear) {EXPECT_EQ(getNumDaysInMonth(3, false), 30);}
TEST(GetNumDaysInMonthTest, MayNonLeapYear) {EXPECT_EQ(getNumDaysInMonth(4, false), 31);}
TEST(GetNumDaysInMonthTest, JunNonLeapYear) {EXPECT_EQ(getNumDaysInMonth(5, false), 30);}
TEST(GetNumDaysInMonthTest, JulNonLeapYear) {EXPECT_EQ(getNumDaysInMonth(6, false), 31);}
TEST(GetNumDaysInMonthTest, AugNonLeapYear) {EXPECT_EQ(getNumDaysInMonth(7, false), 31);}
TEST(GetNumDaysInMonthTest, SepNonLeapYear) {EXPECT_EQ(getNumDaysInMonth(8, false), 30);}
TEST(GetNumDaysInMonthTest, OctNonLeapYear) {EXPECT_EQ(getNumDaysInMonth(9, false), 31);}
TEST(GetNumDaysInMonthTest, NovNonLeapYear) {EXPECT_EQ(getNumDaysInMonth(10, false), 30);}
TEST(GetNumDaysInMonthTest, DecNonLeapYear) {EXPECT_EQ(getNumDaysInMonth(11, false), 31);}
And with that, any change to the function and its data that breaks the functionality of the code is caught by the tests.
Is there any functionality in Mull that can mutate data like defined in the array daysInMonthArray?
With the recent removal of the scalar_value_mutator in PR #1109, there does not seem to be a way to mutate scalar data like defined in an array like daysInMonthArray. Is that true or is there some setting that I am issing?
SIDENOTE: I was starting to look into using LLVM DataFlowSanitizer (DFSan) to detect if data defined in the code under test is being used in the outputs. But unlike Mull, DFSan requires you rebuild all code (including libc++) with support for DFSan or creating wrappers for all functions that are called by your code under test between the data and the outputs. Also, the process to taint the data and check for it in the outputs is manual (you have to edit the source code for the function and test tests). And it can only label up to 8 selections of data. So while it seems that DFSan provides the needed basic functionality to determine if the data in the code under tests is being used to produce the outputs, it seems (unlike Mull) completely impractical to use in production legacy C++ software.
I noticed in:
- #775
the bullet:
- AST-based mutations: after a lot of considerations we decided that AST-based mutations (based on Clang plugins) are the must-have.
I am guessing that is one impediment to being able to mutate scalar data like in the example above?
@AlexDenisov, responding to your comment:
- https://github.com/mull-project/mull/issues/1102#issuecomment-3403369218
here ...
Right now, Mull doesn't have anything that could help finding the issue in the test case. I will need to take a closer look and to re-evaluate how to add better mutators for constants.
Okay, good to get some clarity on that. Just wanted to make sure that I did not miss something.
Do you think that might be possible if Mull adds support for AST-based mutators?
Otherwise, any opinions on trying to use DFSan for the purpose of checking if data defined in the SUT is impacting at least one of the outputs? It seems like it might be able to detect data that does not impact the outputs. But there are a lot of practical issues with DFSan that would make it very hard to use in production (unlike Mull which is much easer to use with a large production code with an existing install of LLVM and related libraries).
Thanks!