Split pending and skipped tests

[just-boris]

This is a first step to solve https://github.com/webdriverio/wdio-allure-reporter/issues/26

Now skipped tests will be reported with skipped status. The pending status is still there for really pending tests (tests with empty implementation).

[just-boris]

~~But the pending tests are still broken. They are firing test:start event that still breaks Allure. I don't know how widely pending feature is used, but we need to find a solution for this as well.~~ Fixed that, now they are all also are saved as skipped for sake of consistency

[just-boris]

:warning: Breaking change: after that fix, all reporters, like spec will stop report skipped steps as pending. To bring the previous behavior back, they should listen to new test:skipped event.

[just-boris]

ping @christian-bromann

[christian-bromann]

To bring the previous behavior back, they should listen to new test:skipped event.

Let's make sure first to have all reporters updated before landing this change.

[esclapes]

I just bumped into this issue. Allure is not able to generate the reports from our cucumber features and this patch seems to fix the issue. As I understand the malformed XML files are missing stop and status attributes. Is there any way we can get those from current events?

How could I help?

[christian-bromann]

@esclapes @just-boris we need to mitigate the inconsistencies between all frameworks and make sure that they all fire the same events

[esclapes]

I have had a look at the different framework adapters to check how/when are they emitting events. There are indeed some inconsistencies. I might have missed something, but here are my two cents:

• there seems to be no test:end event sent in the cucumber adapter. Is that right?
• cucumber seems to have many more status so the bundling makes the abstraction more leaky. Would it hurt to add new states on the wdio side even if they do not map to all frameworks?
• in terms of implementation a more explicit mapping of framework events like the one in MochaAdapter::EVENTS would make it easier to standardize and spot the differences. It is specially difficult to know what is emitted when doing things like var e = 'test:' + test.status.replace(/ed/, '')

This is the overview.

	baseReporter	cucumber	mocha	jasmine	qunit
suite:start	x	Before feature and scenario	suite	suiteStarted	suiteStart
suite:end	x	After feature and scenario	suite end		testStart
test:start	x	Before step	test	specStarted	testStart
test:end	x	-	test end	specDone: any	testEnd
test:fail	x	Handle: FAILED & UNDEFINED	fail	specDone: failed	testEnd: “other”
test:pass	x	Handle: PASSED	pass	specDone: passed	testEnd: passed
test:pending	x	Handle: PENDING, SKIPPED & AMBIGUOUS	pending	specDone: pending	testEnd: pending
test:disabled?	-			specDone: disabled
hook:start	x	-	hook		?
hook:start	x	-	hook end		?