KeyboardInterrupt terminates log non-gracefully

[ktkopone]

After a testsuite is halted due to keyboardinterrupt, the ControllerInfo summary in the yaml log file will have 0 in all categories (even Requested) regardless of prior test results. Also, the current test will have mostly null values recorded, and any unrun testcases will not have entries.

It would be nice if all results received prior to the interrupt were properly recorded. It would also be nice if all the tests that never got to be started were listed as skipped with an appropriate explanation. Finally it's up to you whether or not the current test should have an explicit fail, a skip, or leave it with the nulls, since any of those is a more subjective choice (can the test really be said to 'fail' when it was interrupted? Perhaps not). But it would be nice if the current test log entry mentioned in some way that it stopped due to a keyboard interrupt and not due to its own or Mobly's fault.

[xpconanfan]

Thanks for reporting!

We actually have a plan to improve sigterm handling, including releasing the controller objects properly in this situation.

Would you be willing to tell us more about your use case? It sounds like you are running Mobly with a larger system that expects the records in results to be consistent?

[ktkopone]

We're looking to integrate with Jenkins using XML reporting, for regularly run UI tests. The ideal case likely would have been xmlrunner compatability as a test runner, but I've made a yaml-xml converter which is sufficient for our needs.

The KeyboardInterrupt isn't a particularly expected use case for us, but I happened to run into this while developing the xml converter and figured it merited an issue.

The other use case I found myself wanting (and which might merit another issue) is a way to pass args into the BaseTest class's instantiation or its call to setup_class() - it can grab a var(s) from the config file, but we'd like to be able to override the config value with a cmd line flag as well. Since test_runner keeps the class object to itself, I worked around this with global vars for now, which is quite ugly.

[xpconanfan]

1. How will your CI terminate Mobly tests? I assume it would send SIGTERM? So the real request here is handling SIGTERM correctly? It is difficult to guarantee the existence of records for all tests bc the interrupt can happen at any point and for cases like Android instrumentation runner, we don't know what test cases exist until the execution has happened.

2. So Mobly's reporting is designed for e2e testing. While xml runner reporting is very popular, it cannot satisfy our requirements. How are you massaging Mobly results into the xmlrunner format?

3. For cli arg override, we have this as part of our internal infra, so adding it to Mobly itself has not been a high priority. Can you file a separate request for that :)

[ktkopone]

We simply call test_runner.main() in a larger python script as in some of the examples, with some cli args passed into it via string. test_runner.main() kills itself with sys.exit (if a test failed) when its done, I catch that and then simply convert the yaml log directly to xml as best I can, using python's xml.etree.ElementTree module; not trying to follow the xmlrunner format very closely. If it's not feasible to record test info realtime, that's fine since as mentioned it's an edge case.

I'll open an issue for the cli args passing

[xpconanfan]

Ah, that's good to know.

While test_runner.main() is the default entry point for running Mobly, if you have more advanced needs in pre-run actions like custom cli args that are specific to your internal toolchain, we actually recommend using the TestRunner APIs directly.

We made sure the TestRunner APIs are usable by third-party to create their own suite with custom behaviors. You can consider suite_runner an example of a custom entry point (sorry about the confusing name). It loads the config, adds some test classes to a TestRunner instance, and calls TestRunner.run.

So you can implement your own suite_runner like entry point and parse args however you like or even with extra pre-run/post-run processing steps defined

We didn't advertise this in github bc it has not been asked by external users. Now that we see the usage, we will try to add more docs on that.