cosmic-ray

cosmic-ray copied to clipboard

Currently, the log in the database does not contain any information about the runtime of the individual job_ids. But I think this could be quite an interesting information to have.

The runtime can be logged either as a duration or with two timestamps: start_job and end_job.

I'm not completely opposed to adding this, but I do worry about adding features that don't serve any real purpose. I don't see any immediate benefit to CR of adding this information to the output.

But I'm happy to be convinced. What do you plan to use this information for? How do you see it improving Cosmic Ray?

Permutation testing is slow. And it is generally not enough to run it just once. Hence, I wanted to:

1. narrow the scope of what is permutation tested in my code base
2. reduce the set of applicable permutation operators
3. see the estimate of how much time it will take, if I do the tasks above

It felt natural to look for the answer to the 3rd point it in the log.

Furthermore, if you want to improve something, you generally have to first be able to measure it. Hence, I thought that it could be useful to have runtime measures for further development of Cosmic Ray.

Sorry for the delayed response; I've had to spend some time sorting out CR problems on windows...not my favorite thing to do.

My general approach to estimating testing time has been to simply multiply the runtime of my test suite on unmutated code by the number of mutations. This obviously isn't perfect, but it gives you a good idea.

I won't be able to put any time real towards capturing/storing run times any time soon, but if you want to put some effort into it I'd be happy to help you. Off the top of my head, you'd need to:

• update the database format to store the timing information
• put a timer around the test runs
• communicate the timing information from testing subprocesses to the cosmic-ray exec process.
• Update any relevant reporting tools (e.g. cr-html).

I have projects to work on. If someone else would like to write the PR, go ahead.