mull
mull copied to clipboard
mull-project
On the way to v1.0
This issue is supposed to be a place to discuss what needs to be implemented for version 1.0. These should include show-stoppers but can also include nice-to-haves. We'll need to decide which ones belong to which group.
Everyone is encouraged to share their ideas and suggestions. I'll start.
- [x]
Multithreading support: mull itself is multithreaded, but if the program under test is multithreaded, then Mull may produce non-deterministic results. Currently, Mull gathers the code coverage based on call trees. In a single-threaded program (most of the time) the call tree is deterministic (i.e., the order of the calls is the same from run to run). A possible solution is to use another, 'linear' coverage approach: to only collect whether the function was executed or not. It has some drawbacks but will give the most deterministic behavior. Not needed anymore - [x]
TLS support: this is being in the works currently #745. Not needed anymore - [x] Diff reporter: Mull has several reporters (mutation elements, SQLite, CLI). it would be very nice to also have a reporter that produces git-like patches so that they can be applied to the source code or used as part of CI integration. Related #587.
- [ ] CI integration: mull can be run as part of the CI process, but the integration is not trivial and does not have smooth CI integration. Directly related to #738.
- [x] Diff-based mutation filter: at the moment, Mull does not respect version control systems. For better CI integration it is essential to filter out mutants that are unrelated to a pull or merge request.
- [x] Non covered mutants: as described here #636.
- [x]
More test frameworks: it would be nice to have more test frameworks supported. Related #174 #175 #191 #492 #746. More suggestions are very welcome. Cannot support it anymore - [ ] AST-based mutations: after a lot of considerations we decided that AST-based mutations (based on Clang plugins) are the must-have.
The above list is not exhaustive and may be extended.
- Smooth junk detection: currently, Mull requires a user to provide 'implicit' C/C++ header search paths manually. It would be nice if we can automate this process in a transparent way.
Great to collect all the important features in one ticket. Would it be useful to also create a v1.0 Label or Milestone to mark PRs and issues which are part of this effort?
That's a good idea. I'll create a milestone and a label. I guess for the milestone to work I'd also need to create an issue per each feature/task.
About the diff filter, are we talking something like clang-tidy-diff? https://clang.llvm.org/extra/doxygen/clang-tidy-diff_8py_source.html
About the diff filter, are we talking something like clang-tidy-diff? https://clang.llvm.org/extra/doxygen/clang-tidy-diff_8py_source.html
That is correct.
Vote for "Add ability to catch mutants using tests written on other languages #778"
More test frameworks: it would be nice to have more test frameworks supported. Related #174 #175 #191 #492 #746. More suggestions are very welcome.
It is definitely good for those who use these frameworks. But I believe it would be a nightmare for Mull maintainers to support and test all of these frameworks. I would suggest to generalize integration with test frameworks (no ideas how) or simplify integration that will allow users easily integrate without changing Mull source code.
Also there is a report published by Jetbrains where developers answered on question "Which unit-testing frameworks do you regularly use, if any?". Below is a distribution of different unit testing frameworks for C and C++:
Source: C language:
Source: C++ language
Yet another follow-up: since the JIT is gone there is no special need for multithreading and TLS support. Also, we cannot have/don't need explicit test frameworks support anymore.
I edited the original comment to reflect the current state.