Mull compiler plugin ignores the timeout option in mull.yml

[bartlettroscoe]

Description

It seems that the Mull compiler plugin is ignoring the timeout option specified in the file mull.yml. For example, for the mull.yml file:

mutators:
 - cxx_all
timeout: 1 # 1 millisecond

Just the generation of mutations can take much longer than what is specified in timeout (see demonstration below).

Mull version

Observed with Mull version:

$ mull-runner-19 --version
Mull: Practical mutation testing and fault injection for C and C++
Home: https://github.com/mull-project/mull
Docs: https://mull.readthedocs.io
Support: https://mull.readthedocs.io/en/latest/Support.html
Version: 0.27.0
LLVM: 19.1.7

Workaround

A simple workaround is to use the Linux timeout command as the compiler wrapper, e.g. timeout <max-duration> clang++ .... You can use timeout as the compiler wrapper with your own custom compiler wrappers when using OpenMPI by setting the environment variables:

export OMPI_CC="timeout ${BUILD_TIMEOUT} $(which clang)"
export OMPI_CXX="timeout ${BUILD_TIMEOUT} $(which clang++)"

With that, the compiler invocations time out correctly. But it would be better if the Mull timeout option were respected and compiler invocations timed out as documented.

Demonstration

For a small C++ project I have, numerical_coverage_examples, I can demonstrate this behavior. That project is not open source (yet), but I can also reproduce this with a Trilinos build. I can provide exact reproduction instructions for the latter if needed. (But it should be trivial to reproduce this behavior with any project configured to build with Mull.)

For the project numerical_coverage_examples, I have the mull.yml file:

$ cat /mounted_from_host/numerical_coverage_examples/BUILDS/clang-19-mull-asan/mull.yml
mutators:
 - cxx_all
timeout: 1 # 1 millisecond
includePaths:
  - '.*[\\/]numerical_coverage_examples[\\/].*'
excludePaths:
  - '.*[\\/]external[\\/].*'
  - '.*unittests[.]cpp'
  - '.*unit_tests[.]cpp'
  - '.*TestSuite.*[.]cpp'
  - '.*/CommonUtils_computeAndCheckRelError[.]hpp'
  - '.*/KahanSumTestSupport[.]hpp'
  - '.*/lu_decomp/lu_decomp_example[.]cpp'
  - '.*horner_poly/naivePolyErrorGrowth[.]cpp'
debug:
 slowIRVerification: true
quiet: false # enables additional logging

I then build one of the object files using this mull.yml file with:

$ time make lu_decomp_example.o
Building CXX object lu_decomp/CMakeFiles/lu_decomp_example.dir/lu_decomp_example.cpp.o
[info] Using configuration /mounted_from_host/numerical_coverage_examples/BUILDS/clang-19-mull-asan/mull.yml
[info] Found compilation flags in the input bitcode
[info] Gathering functions under test (threads: 1)
       [################################] 1/1. Finished in 0ms
[info] Instruction selection (threads: 28)
       [################################] 41/41. Finished in 11ms
[info] Searching mutants across functions (threads: 28)
       [################################] 41/41. Finished in 11ms
[info] Applying filter: no debug info (threads: 28)
       [################################] 2466/2466. Finished in 0ms
[info] Applying filter: file path (threads: 28)
       [################################] 2466/2466. Finished in 11ms
[info] Applying filter: manually ignored mutants (threads: 28)
       [################################] 2466/2466. Finished in 10ms
[info] Applying filter: junk (threads: 28)
       [################################] 2434/2434. Finished in 314ms
[info] Prepare mutations (threads: 1)
       [################################] 1/1. Finished in 0ms
[info] Cloning functions for mutation (threads: 1)
       [################################] 1/1. Finished in 229ms
[info] Removing original functions (threads: 1)
       [################################] 1/1. Finished in 10ms
[info] Redirect mutated functions (threads: 1)
       [################################] 1/1. Finished in 1ms
[info] Applying mutations (threads: 1)
       [################################] 414/414. Finished in 26956ms
real    0m38.773s
user    0m38.656s
sys     0m1.196s

If the timeout option worked, the Clang compiler should have stopped well before 38 seconds. The log shows 26.956s spent applying 414 mutations in the Mull compiler plugin. With a timeout of 1ms, this phase should have been interrupted well before 26s.

[bartlettroscoe]

Wait, I just discovered that timeout option impacts the runtime for mull-runner, not the build time for the Mull compiler plugin. So the only way to timeout the builds is to use the timeout command as a compiler wrapper as per above?

But why then does mull-runner have a --timeout command-line option? So why would there also be a timeout option in the mull.yml file? All of the other options in mull.yml seem to be impacting the Mull compiler plugin. (That is understandable because it is hard to pass options to a compiler plugin in a targeted way but easier to pass them to mull-runner in the command-line argument list.)

[bartlettroscoe]

NOTE: Adding to the confusion, the must recent update to the documentation:

• https://github.com/mull-project/mull/blob/e13a24576192cfb553b56a312f76195a9cd3272e/docs/MullConfig.rst

says:

Mull's IR frontend is configured via a text file in the yaml format.

There is no mention of this file mull.yml impacting the runtime behavior of other Mull tools like mull-runner. And sense you can directly call mull-runner with your own command-line arguments and since mull-runner supports a --timeout option, why does mull.yml even support a timeout option?

But looking at the parsing code for mull.yml at:

https://github.com/mull-project/mull/blob/1bd12f6f68336489b071cf549dcdc1fd670a7790/lib/Config/ConfigurationParser.cpp#L42

it does look like there are other options that impact mull-runner like captureTestOutput, captureMutantOutput, and includeNotCovered, which also have matching command-line options in mull-runner.

However, I can't find any documentation that directly says that mull-runner is impacted by the mull.yml file (or the MULL_CONFIG env var pointing to a config file).

So this issue is likely just a misunderstanding related to some missing Mull documentation?

In any case, it would seem that the Mull compiler plugin does not support a timeout feature, so users will need to implement their own compile timeouts using the workarounds given above?