Optimize DB Disk Usage

[jtpereyda]

1. Done: ~Stop saving to the DB every single step. Maybe at test case close?~
2. Done: ~Save the DB when high-level exceptions are caught.~
3. Optional: See if DB save can be asynchronous.

[SR4ven]

I'm currently working on a solution that keeps track of the last n test cases and only saves them to disk when a error or failure is logged. Saving the log on every test case close would help reducing disk I/O but it would still mean hundreds of thousands of writes for large projects. That's way too much for an SSD or SD card long term. For me there's no point saving good test cases but rather only the bad ones and maybe a few cases before the crash just to be on the safe side. I will put a switch in so you can decide what to log.

[jtpereyda]

Some other reasons you might want to save the data sooner:

1. Keep track of progress if the test gets stopped.
2. Keep a record of behavior for detailed analysis afterward.

That said, any optimization is likely to be an improvement over where it is now.

[SR4ven]

1. When you stop the test by SIGINT it will trigger log_error which will cause the db logger to save the last n test cases. Would that be sufficient? It wouldn't save when pausing the run thru the webinterface.
2. What exactly do you mean by that? Would this result in keeping all log cases but only writing them to disk at completion or error/fail? That could be done be setting the "save last n test cases" really high, so no test case logs are being dropped.

[jtpereyda]

When you stop the test by SIGINT it will trigger log_error which will cause the db logger to save the last n test cases. Would that be sufficient? It wouldn't save when pausing the run thru the webinterface.

That approach makes sense.

2. Keep a record of behavior for detailed analysis afterward. 2. What exactly do you mean by that?

I was just thinking out loud on the reasons to keep a record even if 100% of test cases passed. I don't know that it requires any change to your plan. It sort of depends on user preference -- one user might want to run millions of test cases and not keep them all (I think this is what you mean by saving only n test cases prior to a failure). Another user or situation might call for preemptively saving all of the "passed" test cases in case some subtle undetected error comes up later. Just thinking through the use cases.

[SR4ven]

1. Stop saving to the DB every single step. Maybe at test case close?

2. Save the DB when high-level exceptions are caught.

3. Optional: See if DB save can be asynchronous.

1. and 2. are completed with #233