Wip: tests in draft mode

[jbrazile]

A colleague and I would like selenese-runner-java to replace our proprietary in-house tool. Thanks for recently merging the "retry" feature. Other than that feature, the most important missing feature is what our users call "tests in draft mode". Often tests are implemented before the application-under-test is complete. It is useful to still execute those tests and monitor their results in a continuous build, but to acknowledge that such tests can fail and therefore shouldn't cause the build to fail.

This patch attempts to illustrate the desired feature. Thanks in advance for considering such a feature - even if it isn't this particular implementation: A few notes regarding it:

• maybe someone has a better idea of how to mark a test as DRAFT (rather than adding a special comment) - but we'd like it to be easy for test authors using an IDE like katalon or selenium-ide
• we looked at the recent "flakyError" and "flakyFailure" attributes (https://maven.apache.org/surefire/maven-failsafe-plugin/examples/rerun-failing-tests.html) but only "skipped" seems to be currently understood by the current Jenkins "publish as JUnitTest" feature, so we used "skipped".
• in our local tool, results from DRAFT tests are "greyed-out" to distinguish from passed ("green") and failed ("red") - but we haven't proposed any such changes to the existing selenese-runner-java HTML report - if you would like, we could take a look at that and make a suggestion.
• we've added a few junit tests here to test this implementation and will try to monitor the github pull request build in case breakages need to be corrected.

[vmi]

Hi,

Thank you for your PR.

It's looks good. However, I think that there is room for discussion about enum Lifecycle and draft-mode marker string.

• enum Lifecycle

If you can not think of more than 2 Lifecycle states, I think that boolean is better as follows:

if (testCase.isDraftMode()) { ... }

• draft-mode marker string

I propose magic comments of the Emacs format as follows:

-*- lifecycle: draft -*-

[vmi]

or: -*- draft-mode: true -*-

[jbrazile]

One of our users made a very interesting counter-proposal.

The current proprietary tool (which I hope to replace with selenese-runner-java) allows "pragmas" for signalling DesiredCapabilities.

Appium seems to have added some extensions to this

Can you imagine a comment syntax for these - perhaps with the addition of this "draft-mode"?

Edit: If yes, I could include such a pragma-parser in this pull-request, or make a separate pull-request for that feature including an attempt at implementing DesiredCapabilities handling.

Edit 2: Here is browserStack's view of DesiredCapabilities

[jbrazile]

Just to be clear, I acknowledge the concept of draft-mode is separate/orthogonal from DesiredCapabilities, so it wouldn't bother me if we don't merge these two into one conceptual mechanism (even if the implementation might be largely the same).