mull Extend the Mull compiler plugin to accept a list of file paths and line blocks to restrict mutations

Description

The Mull compiler plugin currently only supports restricting which lines of code get mutated on a per-file basis with the includePaths list in the mull.yml file, or by using a Git diff with the gitDiffRef option in the mull.yml file. This does not provide sufficient flexibility for specific use cases like:

Iteratively rebuilding to debug and squash a small number of surviving mutations
Focusing on generating tests for a single function in a large source file

For use cases like that, you want the ability to tell the Mull compiler plugin the exact blocks of code in which to generate mutations. Without that ability, iterative rebuilding and rerunning a single test executable can take over 20 minutes, even for moderately sized C++ code.

For the example use case below, being able to focus the mutations would reduce build time from over 20 minutes to just 45 seconds while still providing all the value we need. This is summarized in the table:

Description	# mutations generated	wall-clock time
No mutations	0	30s
Mutations of only function of interest using gitDiffRef workaround	20	45s
Mutation of entire file using includePaths	825	20m

We need a direct and clean solution to this problem, not a workaround that relies on git diff.

Proposed solution

The proposed solution is to define a file that the Mull compiler plugin will read, with the following format:

<file-path-1>:<start-line-1>-<end-line-1>
<file-path-1>:<start-line-2>-<end-line-2>
...
<file-path-2>:<start-line-1>-<end-line-1>
...

This file could be referenced by the mull.yml file with a new option, for example:

restrictMutationsToFilesAndLines: <file-path>

The internal implementation would augment the same data structure that the current gitDiffRef feature augments, as shown at:

https://github.com/mull-project/mull/blob/9206c0b857e3433fedeef3dfe7f7fea096e425ff/lib/Filters/Filters.cpp#L94
https://github.com/mull-project/mull/blob/9206c0b857e3433fedeef3dfe7f7fea096e425ff/lib/Filters/GitDiffFilter.cpp#L18
https://github.com/mull-project/mull/blob/9206c0b857e3433fedeef3dfe7f7fea096e425ff/lib/Filters/GitDiffReader.cpp#L16
https://github.com/mull-project/mull/blob/9206c0b857e3433fedeef3dfe7f7fea096e425ff/lib/Filters/GitDiffReader.cpp#L36
https://github.com/mull-project/mull/blob/9206c0b857e3433fedeef3dfe7f7fea096e425ff/lib/Filters/Filters.cpp#L99
https://github.com/mull-project/mull/blob/9206c0b857e3433fedeef3dfe7f7fea096e425ff/lib/Driver.cpp#L168

So the basic functionality to restrict mutations to a specific set of files and specific blocks of code in those files is already supported by Mull. We just need to provide an API to pass that information into Mull (indirectly) through the mull.yml file.

Demonstration

To demonstrate the issues involved, consider the use case where you are extracting a function from a large C++ file for the Trilinos project. Here, I will use a specific branch in my mirror of Trilinos:

https://github.com/bartlettroscoe/MyTrilinos/tree/tpetra-crsmatrix-ef-codex-gpt-oss-120b-medium-2025-10-22_2025-10-29

(NOTE: I can provide exact reproduction instructions using the container image bartlettroscoe/mull-trilinos-clang-19.1.6-openmpi-4.1.6:2025-10-21 if anyone is interested to reproduce.)

First, consider building a single *.o file while mutating only the code in the following source files:

https://github.com/bartlettroscoe/MyTrilinos/blob/tpetra-crsmatrix-ef-codex-gpt-oss-120b-medium-2025-10-22_2025-10-29/packages/tpetra/core/src/Tpetra_CrsMatrix_decl.hpp
https://github.com/bartlettroscoe/MyTrilinos/blob/tpetra-crsmatrix-ef-codex-gpt-oss-120b-medium-2025-10-22_2025-10-29/packages/tpetra/core/src/Tpetra_CrsMatrix_def.hpp

We set up Mull to put mutations only in those files via the mull.yml configuration:

mutators:
 - cxx_all
ignoreMutators:
#  - cxx_logical_and_to_or
timeout: 3480000 # 58 minutes (in milliseconds)
includePaths:
#  - '.*[\\/]Trilinos[\\/].*'
  - '.*/Trilinos/packages/tpetra/core/src/Tpetra_CrsMatrix_de.*[.]hpp'
excludePaths:
  - '.*[\\/]gtest[\\/].*'
  - '.*[\\/]test[\\/].*'
  - '.*[\\/]UnitTest[\\/].*'
  - '.*[\\/]UnitTesting[\\/].*'
  - '.*[\\/]unit_tests[\\/].*'
  - '.*[\\/]unit_test[\\/].*'
  - '.*[\\/]performance_tests[\\/].*'
  - '.*[\\/]perf_test[\\/].*'
  - '.*[\\/]compile_tests[\\/].*'
  - '.*[\\/]example[\\/].*'

After configuring, having Mull mutate just that one file takes over 20 minutes:

$ export CCACHE_DISABLE=1

$ touch /mounted_from_host/Trilinos/packages/tpetra/core/src/Tpetra_CrsMatrix_def.hpp

$ time ninja packages/tpetra/core/src/CMakeFiles/tpetra.dir/Tpetra_CrsMatrix_DOUBLE_INT_LONG_LONG_SERIAL.cpp.o

[1/1] Building CXX object packages/tpetra/core/src/CMakeFiles/tpetra.dir/Tpetra_CrsMatrix_DOUBLE_INT_LONG_LONG_SERIAL.cpp.o
[info] Using configuration /mounted_from_host/Trilinos/BUILDS/clang-19-genconfig-mull/mull.yml
[info] Found compilation flags in the input bitcode
[info] Gathering functions under test (threads: 1)
       [################################] 1/1. Finished in 1ms
[info] Instruction selection (threads: 28)
       [################################] 231/231. Finished in 75ms
[info] Searching mutants across functions (threads: 28)
       [################################] 231/231. Finished in 71ms
[info] Applying filter: no debug info (threads: 28)
       [################################] 28848/28848. Finished in 11ms
[info] Applying filter: file path (threads: 28)
       [################################] 28848/28848. Finished in 20ms
[info] Applying filter: manually ignored mutants (threads: 28)
       [################################] 28848/28848. Finished in 21ms
[info] Applying filter: junk (threads: 28)
       [################################] 28848/28848. Finished in 8875ms
[info] Prepare mutations (threads: 1)
       [################################] 1/1. Finished in 0ms
[info] Cloning functions for mutation (threads: 1)
       [################################] 1/1. Finished in 6362ms
[info] Removing original functions (threads: 1)
       [################################] 1/1. Finished in 537ms
[info] Redirect mutated functions (threads: 1)
       [################################] 1/1. Finished in 1ms
[info] Applying mutations (threads: 1)
       [################################] 825/825. Finished in 1064380ms

real    20m59.211s
user    21m0.338s
sys     1m38.148s

Furthermore, two other instantiations of the templates in those files are needed to build and link a single unit test executable after that object file is built:

$ cd packages/tpetra/core/test/CrsMatrix/

$ time make TpetraCore_CrsMatrix_UnitTests
...
[446/446] Linking CXX executable packages/tpetra/core/test/CrsMatrix/TpetraCore_CrsMatrix_UnitTests.exe

real    22m52.299s
user    79m6.067s
sys     6m4.231s

Taking almost 25 minutes to do a single rebuild so we can rerun a single test binary with mull-runner is obviously unacceptable. That kills developer productivity.

Suppose we are trying to test the extraction of the new function transferAndFillComplete_getCallersParameters() as seen in the commit:

https://github.com/bartlettroscoe/MyTrilinos/commit/446242a7df505beb5394ce0da113b555607e9934

and we really only want Mull to add mutations for the lines of code within the new function:

https://github.com/bartlettroscoe/MyTrilinos/blob/446242a7df505beb5394ce0da113b555607e9934/packages/tpetra/core/src/Tpetra_CrsMatrix_def.hpp#L8709-L8743

Since Mull does not have a feature to do this directly, we can fake it by adding a newline to the end of each line that we want Mull to mutate so those lines show up in the diff:

$ git diff --word-diff-regex=.
diff --git a/packages/tpetra/core/src/Tpetra_CrsMatrix_def.hpp b/packages/tpetra/core/src/Tpetra_CrsMatrix_def.hpp
index 0a23fd8c909..b29cf696d56 100644
--- a/packages/tpetra/core/src/Tpetra_CrsMatrix_def.hpp
+++ b/packages/tpetra/core/src/Tpetra_CrsMatrix_def.hpp
@@ -8706,41 +8706,41 @@ CrsMatrix<Scalar, LocalOrdinal, GlobalOrdinal, Node>::
transferAndFillComplete_getCallersParameters(
    const Teuchos::RCP<Teuchos::ParameterList>& params,
    const ::Tpetra::Details::Transfer<LocalOrdinal, GlobalOrdinal, Node>& rowTransfer) const {
  TransferAndFillCompleteParams p; 
 {+ +}// Initialize defaults (matching original code) 
 {+ +}p.isMM = false; // optimize for matrix-matrix ops. 
 {+ +}p.reverseMode = false; // Are we in reverse mode? 
 {+ +}p.restrictComm = false; // Do we need to restrict the communicator? 
 {+ +}p.mm_optimization_core_count = Details::Behavior::TAFC_OptimizationCoreCount(); 
 {+ +}p.matrixparams = Teuchos::null; 
 {+ +}p.overrideAllreduce = false; 
 {+ +}p.useKokkosPath = false; 
 {+ +}if (!params.is_null()) { 
   {+ +}p.matrixparams = sublist(params, "CrsMatrix"); 
   {+ +}p.reverseMode = params->get("Reverse Mode", p.reverseMode); 
   {+ +}p.useKokkosPath = params->get("TAFC: use kokkos path", p.useKokkosPath); 
   {+ +}p.restrictComm = params->get("Restrict Communicator", p.restrictComm); 
   {+ +}auto& slist = params->sublist("matrixmatrix: kernel params", false); 
   {+ +}p.isMM = slist.get("isMatrixMatrix_TransferAndFillComplete", false); 
   {+ +}p.mm_optimization_core_count = slist.get("MM_TAFC_OptimizationCoreCount", p.mm_optimization_core_count); 
   {+ +}p.overrideAllreduce = slist.get("MM_TAFC_OverrideAllreduceCheck", false); 
   {+ +}if (getComm()->getSize() < p.mm_optimization_core_count && p.isMM) p.isMM = false; 
   {+ +}if (p.reverseMode) p.isMM = false; 
 {+ +}}  
{+  +}
{+  +}// Only used in the sparse matrix-matrix multiply (isMM) case. 
 {+ +}p.iallreduceRequest.reset(); 
 {+ +}p.mismatch = 0; 
 {+ +}p.reduced_mismatch = 0; 
 {+ +}if (p.isMM && !p.overrideAllreduce) { 
   {+ +}const bool source_vals = !getGraph()->getImporter().is_null(); 
   {+ +}const bool target_vals = !(rowTransfer.getExportLIDs().size() == 0 || 
                              {+ +}rowTransfer.getRemoteLIDs().size() == 0); 
   {+ +}p.mismatch = (source_vals != target_vals) ? 1 : 0; 
 {+ +}p.iallreduceRequest = ::Tpetra::Details::iallreduce(p.mismatch, p.reduced_mismatch, 
                                {+ +}Teuchos::REDUCE_MAX, *(getComm())); 
 {+ +}} 
 {+ +}return p;{+ +}
}

And with the following mull.yml settings:

includePaths:
  - '.*[\\/]Trilinos[\\/].*'
excludePaths:
  - '.*[\\/]gtest[\\/].*'
  - '.*[\\/]test[\\/].*'
  - '.*[\\/]UnitTest[\\/].*'
  - '.*[\\/]UnitTesting[\\/].*'
  - '.*[\\/]unit_tests[\\/].*'
  - '.*[\\/]unit_test[\\/].*'
  - '.*[\\/]performance_tests[\\/].*'
  - '.*[\\/]perf_test[\\/].*'
  - '.*[\\/]compile_tests[\\/].*'
  - '.*[\\/]example[\\/].*'
gitDiffRef: "HEAD"
gitProjectRoot: /mounted_from_host/Trilinos

run the build again:

$ touch /mounted_from_host/Trilinos/packages/tpetra/core/src/Tpetra_CrsMatrix_def.hpp

$ time ninja packages/tpetra/core/src/CMakeFiles/tpetra.dir/Tpetra_CrsMatrix_DOUBLE_INT_LONG_LONG_SERIAL.cpp.o
[1/1] Building CXX object packages/tpetra/core/src/CMakeFiles/tpetra.dir/Tpetra_CrsMatrix_DOUBLE_INT_LONG_LONG_SERIAL.cpp.o
...
[info] Using configuration /mounted_from_host/Trilinos/BUILDS/clang-19-genconfig-mull/mull.yml
[info] Incremental testing using Git Diff is enabled.
- Git ref: HEAD
- Git project root: /mounted_from_host/Trilinos
[info] Found compilation flags in the input bitcode
[info] Gathering functions under test (threads: 1)
       [################################] 1/1. Finished in 1ms
[info] Instruction selection (threads: 28)
       [################################] 10937/10937. Finished in 663ms
[info] Searching mutants across functions (threads: 28)
       [################################] 10937/10937. Finished in 11ms
[info] Applying filter: no debug info (threads: 28)
       [################################] 221/221. Finished in 1ms
[info] Applying filter: file path (threads: 28)
       [################################] 221/221. Finished in 1ms
[info] Applying filter: manually ignored mutants (threads: 28)
       [################################] 221/221. Finished in 10ms
[info] Applying filter: junk (threads: 28)
       [################################] 221/221. Finished in 5168ms
[info] Prepare mutations (threads: 1)
       [################################] 1/1. Finished in 0ms
[info] Cloning functions for mutation (threads: 1)
       [################################] 1/1. Finished in 126ms
[info] Removing original functions (threads: 1)
       [################################] 1/1. Finished in 6ms
[info] Redirect mutated functions (threads: 1)
       [################################] 1/1. Finished in 0ms
[info] Applying mutations (threads: 1)
       [################################] 20/20. Finished in 5197ms

real    0m44.473s
user    0m44.565s
sys     0m8.104s

This workaround reduces the number of mutations that Mull generates from 825 to just 20 and reduces the build time for that one object file from 20 minutes to just 45 seconds. Note that the unmutated build of that object file takes about 30 seconds, so going from 30 to 45 seconds to add Mull mutations for the code we care about is a very small price to pay for this valuable information. And going from 20 minutes to 45 seconds is obviously a game-changer for using Mull in cases like this.

But for real use cases, we can't easily rely on this gitDiffRef workaround. We would have to adopt a workflow where we do a dummy Git commit, then add newlines to the end of each line we want to mutate, and then run the Mull build as above. That complicates workflows involving Mull tremendously and can make it impractical to use in cases like this (where Mull may be very valuable).

Oct 30 '25 15:10 bartlettroscoe

@AlexDenisov, when we get around to using Mull for this use case, we would be happy to implement this along with automated tests and documentation. We would just need a design and UI review and approval before getting started.

As described above, the Mull compiler plugin already has all of the core functionality for this due to the gitDiffRef feature. We just need a very thin UI to pass that data to the Mull compiler plugin to add new entries to the std::vector like shown at:

https://github.com/mull-project/mull/blob/9206c0b857e3433fedeef3dfe7f7fea096e425ff/lib/Filters/Filters.cpp#L99

Oct 30 '25 15:10 bartlettroscoe

I think I'd prefer a new filter not to mix two different features in one place, and I think coverage filter might be a better starting point as it looks similar to what you are looking for, but either one would work I think.

Overall, I'd be glad to see this feature implemented, it looks very useful!

The docs on how to build and hack on mull are outdated as we've moved away from cmake, but you can pick one of the devcontainers, e.g.

devcontainer up --workspace-folder . --config ./.devcontainer/ubuntu_22.04-llvm-15/devcontainer.json
devcontainer exec --workspace-folder . --config ./.devcontainer/ubuntu_22.04-llvm-15/devcontainer.json bash

or any alternative listed here https://containers.dev/supporting.

The containers should be ready to go, simply run

bazel build ...
bazel test ...

to build and test everything.

Please, let me know if you hit any snags, happy to assist.

Oct 30 '25 19:10 AlexDenisov

The other workaround that also requires temp modifications to the source file is to put in mull-off and mull-on comments. For the Tpetra_CrsMatrix_def.hpp example above, you get the same mutations as the gitDiffRef workaround by using the mull.yml file:

mutators:
 - cxx_all
ignoreMutators:
#  - cxx_logical_and_to_or
timeout: 3480000 # 58 minutes (in milliseconds)
includePaths:
#  - '.*[\\/]Trilinos[\\/].*'
#  - '.*/Trilinos/pacakges/teuchos/core/Teuchos_RCP[.].pp'
  - '.*/Trilinos/packages/tpetra/core/src/Tpetra_CrsMatrix_def[.]hpp'
excludePaths:
#  - '.*[\\/]Trilinos[\\/].*'
  - '.*[\\/]gtest[\\/].*'
  - '.*[\\/]test[\\/].*'
  - '.*[\\/]UnitTest[\\/].*'
  - '.*[\\/]UnitTesting[\\/].*'
  - '.*[\\/]unit_tests[\\/].*'
  - '.*[\\/]unit_test[\\/].*'
  - '.*[\\/]performance_tests[\\/].*'
  - '.*[\\/]perf_test[\\/].*'
  - '.*[\\/]compile_tests[\\/].*'
  - '.*[\\/]example[\\/].*'

and the inserted mull-off and mull=on comments:

diff --git a/packages/tpetra/core/src/Tpetra_CrsMatrix_def.hpp b/packages/tpetra/core/src/Tpetra_CrsMatrix_def.hpp
index 0a23fd8c909..45feeb22245 100644
--- a/packages/tpetra/core/src/Tpetra_CrsMatrix_def.hpp
+++ b/packages/tpetra/core/src/Tpetra_CrsMatrix_def.hpp
@@ -1,3 +1,4 @@
+// mull-off
 // @HEADER
 // *****************************************************************************
 //          Tpetra: Templated Linear Algebra Services Package
@@ -8706,6 +8707,7 @@ CrsMatrix<Scalar, LocalOrdinal, GlobalOrdinal, Node>::
 transferAndFillComplete_getCallersParameters(
     const Teuchos::RCP<Teuchos::ParameterList>& params,
     const ::Tpetra::Details::Transfer<LocalOrdinal, GlobalOrdinal, Node>& rowTransfer) const {
+  // mull-on
   TransferAndFillCompleteParams p;
   // Initialize defaults (matching original code)
   p.isMM = false; // optimize for matrix-matrix ops.
@@ -8741,6 +8743,7 @@ transferAndFillComplete_getCallersParameters(
                                  Teuchos::REDUCE_MAX, *(getComm()));
   }
   return p;
+// mull-off
 }
 
 template <class Scalar, class LocalOrdinal, class GlobalOrdinal, class Node>

And building:

$ export CCACHE_DISABLE=1

$ time ninja packages/tpetra/core/src/CMakeFiles/tpetra.dir/Tpetra_CrsMatrix_DOUBLE_INT_LONG_LONG_SERIAL.cpp.o
[1/1] Building CXX object packages/tpetra/core/src/CMakeFiles/tpetra.dir/Tpetra_CrsMatrix_DOUBLE_INT_LONG_LONG_SERIAL.cpp.o
[info] Using configuration /mounted_from_host/Trilinos/BUILDS/clang-19-genconfig-mull/mull.yml
[info] Found compilation flags in the input bitcode
[info] Gathering functions under test (threads: 1)
       [################################] 1/1. Finished in 1ms
[info] Instruction selection (threads: 28)
       [################################] 199/199. Finished in 84ms
[info] Searching mutants across functions (threads: 28)
       [################################] 199/199. Finished in 81ms
[info] Applying filter: no debug info (threads: 28)
       [################################] 28593/28593. Finished in 1ms
[info] Applying filter: file path (threads: 28)
       [################################] 28593/28593. Finished in 11ms
[info] Applying filter: manually ignored mutants (threads: 28)
       [################################] 28537/28537. Finished in 20ms
[info] Applying filter: junk (threads: 28)
       [################################] 221/221. Finished in 5100ms
[info] Prepare mutations (threads: 1)
       [################################] 1/1. Finished in 0ms
[info] Cloning functions for mutation (threads: 1)
       [################################] 1/1. Finished in 128ms
[info] Removing original functions (threads: 1)
       [################################] 1/1. Finished in 7ms
[info] Redirect mutated functions (threads: 1)
       [################################] 1/1. Finished in 0ms
[info] Applying mutations (threads: 1)
       [################################] 20/20. Finished in 5181ms

real    0m43.706s
user    0m43.211s
sys     0m2.437s

That generated 20 mutations as well, so it was equivalent.

But it would be better for our workflows to not have to modify the source files to to specify the desired lines to be mutated.

Oct 30 '25 19:10 bartlettroscoe

I think I'd prefer a new filter not to mix two different features in one place, and I think coverage filter might be a better starting point as it looks similar to what you are looking for, but either one would work I think.

@AlexDenisov, the difference between this and coverage, is that coverage is generated at runtime and limits the mutations that are run at runtime with mull-runner --coverage-info=<coverage>.profdata ... while we are looking to limit the mutations that are generated at build-time. And we know before we build what lines we want to mutate, so it is critical to limit the mutations at build time (since building object file with all the mutations takes over 20 minutes in our example above).

So what UI would you suggest? Are you okay with adding a new option to the mull.yml file:

restrictMutationToFilesAndLines: <file-path>

?

As the user, I want to be able to provide a file with the list:

<file-path-1>:<start-line-1>-<end-line-1>
<file-path-1>:<start-line-2>-<end-line-2>
...
<file-path-2>:<start-line-1>-<end-line-1>
...

So for my demonstration above, this would just be:

/mounted_from_host/Trilinos/packages/tpetra/core/src/Tpetra_CrsMatrix_def.hpp:8709-8743

You could also consider specifying multiple blocks of lines for the same file like:

<file-path-1>:<start-line-1>-<end-line-1>,<start-line-2>-<end-line-2>,...
...
<file-path-2>:<start-line-1>-<end-line-1>

but I think I would prefer to keep it simple and just list a single range of source-code lines on a single line like above.

Do you agree?

Oct 30 '25 21:10 bartlettroscoe

Agreed, the proposed UI looks good! I think we are on the same page and I like proposed solution

Oct 30 '25 21:10 AlexDenisov