mutant
mutant copied to clipboard
mbj
Add machine readable report
For various cases a machine readable output in a defined JSON (or YAML) format will be handy.
There already exist private non open-sourced implementations of these reports we can maybe opensource.
References #369.
SimpleCov has a standard output, would it be possible to generate that?
You could hookup to code climate
- and coveralls has information about multiple custom formatters together.
Looks like CodeClimate may be their view of the future, and they probably a good standard to follow. Wonder how that changes their current status of code coverage.
Tools like rubocop do have a formatter that looks usable. Not sure if the other tools in the metric_fu suite have similar formatters. I'm betting that code climate is probably getting these tools to produce a common output. Either that or they have given up on them.
@kbrock The simplecov output is made for line based coverage tools, it does not feature the amount of granularity mutant supports. Both for actual coverage and metadata.
I see myself supporting that format as an aggregate, a lossfull conversation to plug into existing ecosystems and also code climate. But I consider this format a bad choice for mutants primary machine readable report format.
@mbj agreed - it is lossy. but it is a very common format for others. Would be great to get mutant code coverate into CodeClimate and/or travis.
I'll say this again, take a look at CodeClimate's format. The spec describes column/offset and length. Their issue construct seems to fit mutant's use cases.
Specifically, I'd like the final output to be aggregated when multiple mutations point to the same problem. E.g. This code block could be self, nil, deleted, ... - which all seem to say, this code is useless or not tested.
I keep throwing around the idea of using sqlite as the interim format. I guess that is kinda the same thing as using json. But I guess the concept of aggregating similar errors together is high on my mind. (including errors / code coverage across multiple reporting engines).
mutant2 does this with configuration files. It seems very cool to me.
Apple products (e.g.: iphoto) went from using text files (xml) to sqlite when they had performance issues. The sqlite is easier to use than the xml file. Probably not saying much as I always have trouble understanding why they design their config files the way they do. But the iphoto database file is not optimal. Sure not as simple as most webapps and their postgres/mysql schema.
What about using the junit XML format? yeah, I know it's XML (yuk), but it is supported by a few CI systems that we use at $work. You could always define extra elements for extra information that isn't part of the actual standard so it would still be consumable by CircleCI, but provide extra mutant-specific details if needed.
I'll say this again, take a look at CodeClimate's format. The spec describes column/offset and length. Their issue construct seems to fit mutant's use cases.
I never voted against supporting this. I just think it would be better to produce a mutant compatible format at mutant domain granularity first.
My special need is to have such a format for another form of incremental mutation testing.
Supporting other formats, including the CodeClimate one, write only is getting my support. Through not prioritization.
Wanted to add a couple wishlist items to the machine readable output at the method level.
- Number of mutations by method. This shows on the interactive display when run in console, but is not (AFAICT) anywhere in the logging when I redirect to a file. It is useful for metrics purposes, as I see it as a useful complexity measure.
- Indicator whether a method has targeted specs. It would be useful to know this if the method is public and has no describe block(s) that specifically target it, so that these can be attacked first to speed mutation process (reduce unnecessary spec targeting) and greatly reduce the number of mutations quickly so as to focus on useful ones. For private methods this would be useful as well, since they can still be targeted by a describe block even without being called directly in the spec. As is known a method without specs often gets a whole series of raise, nil, self, etc. mutations that a straightforward targeted spec can significantly reduce.
The above are helpful as I tend to use mutant beyond just a console tool and more as a frontline critical aspect of my assessment of where to spend limited time budget.
This is a little out there, but what about using sqlite as a report format?
You can write pretty darn quickly to the format and it can be parsed in most languages, as most ruby devs know sql a little.
It makes correlating duplicates (multiple messages per method line) pretty easy. Maybe even correlating across multiple runs.
Got this idea because mongrel2 stores their config files in sqlite. At first it seemed dumb, but after thinking, it made a bunch of sense.
Is the goal to produce a log file that other systems can use out of the box? Or is the thought that the log will be internally focused, with log adapters to produce what ever a CI system would want?
Is the goal to produce a log file that other systems can use out of the box? Or is the thought that the log will be internally focused, with log adapters to produce what ever a CI system would want?
Both. First a mutant secific format. Than information reducing aggregates, to plug into existing ecosystems.
I could be way off base, but using a database would not be the best when running mutant distributed across multiple systems for me - at least I don't think so. It would be easier to take the mutant machine readable (an hopefully human readable) output and post it to a queue to be accumulated.
@mbj Is this feature still on the roadmap? I wanted to integrate mutant with https://github.com/prontolabs/pronto so I need some kind of structured report to pass messages to the tool.
@mknapik This project has no roadmap, and is only driven by my private interest (that is not focused on ruby recently) or commercial needs.
I'm open to accept contributions, and I recently was contacted by https://github.com/joneshf in https://github.com/mbj/mutant/issues/721 who plans something similar.
If you need a guarnatee this feature lands there is also the option to contract me to implement it. For mutant features I offer fixed prices at a reduced rate, and this way other features where already successfully implemented, both for this public version and the private forks.
@mbj Thanks for fast answer. I'm developing pronto-mutant for educational purposes so I'm not interested in hiring anyone. I was thinking about implementing a reporter as part as a new gem but it probably makes more sense to put it in mutant itself for the benefit of all mutant users. Although, I prefer to ask maintainer first to avoid waste of time or duplicating work.
It's good to know the feature is still on the radar. I'll contact @joneshf and see if I can help.
I haven't got a plan of action here yet, nor do I want to take precedence over your contributions. Feel free to work however. @mbj gave some great advice in the #mutant channel on slack: https://mutation-testing.slack.com/archives/C0A0RSER3/p1499706302206963.
The work @mknapik and @joneshf plan to do intersects only slightly but touches the same subsystem Mutnat::Result* as input. Hence I think there is some chance for me only having to explain certain things once.