Allow regex matching for specifying test cases

[puradox]

Summary

The --test_case CLI option currently only supports specifying full test names. This makes it tedious to run only a portion of a large test class (100+ tests) where there are many similar test names, such as when test cases are generated with BaseTestClass.generate_tests.

This issue discusses the design of adding regex matching capability for selecting which tests to run. Is this functionality something we want to include in BaseTestClass for every Mobly user? Or should the user implement this for their own base test class? What should this look like? I propose we add a new CLI option which allows for test case regex matching:

python FooBarTest.py --test_case_regex 'test_(foo|bar)'

Proposal

Add a new CLI option named --test_case_regex (also aliased to --tests_regex, akin to --tests) that performs a full match on test names using regex syntax defined by Python Standard Library re.

• Using a new CLI option benefits the user by being backwards compatible with --test_case and allowing for using both options in conjunction.
• Performing only full matches enforces full specification of test names and behaves more predictably since partial matches are not possible (regex test_can doesn't match test_cannot). In addition, this also removes the need to specify anchors (e.g. ^, $), making regexes more readable (^test_foo_.+$ vs test_foo_.+).
• Using Python regex syntax should be familiar to most users as it is similar to Perl regex. It implements all common regex and is easy to add to Mobly :)

--test_case_regex can be repeated to allow for matching on multiple patterns. This option can also work in conjunction with the --list_tests option; tests matched the regex will be listed rather than ran.

Alternatives explored

Modification of the --test_case option was considered instead of adding a new option. However, there is no guarantee that test names contain regex special characters. If the existing option was to be changed, it would be a breaking change.

I've considered using fnmatch as the pattern matching syntax, similar to ACTS' test_runner, but I've found this syntax confusing since test names are being matched against, not file names. For example test_* works as expected but test_(a|b) doesn't.

Feedback

Let me know what you think!

[xpconanfan]

We intentionally kept complex test case selection outside of the core Mobly runner/base.

In the Mobly repo, we provide mobly/suite_runner.py, which is an example implementation of suite runner.

This way, each user can choose to implement their own runner with their own selection logic/syntax, using APIs provided by Mobly's test_runner and base_test.

So I think this can be added in the mobly/suite_runner.py layer.

[xpconanfan]

I have made several attempts to implement this. Because we don't know all the test cases until a class is being executed (because of generated tests), it's not feasible to do the matching at the suite level.

So looks like this will have to be done in base_test.

I'm still looking at this.