Xcode doesn't show correct test results

[mfclarke]

Please see screenshot. When a test fails, the console reports the correct results (failure), but Xcode (6.3.2) shows green ticks. At first I thought this was because the code being tested raised an exception, but it also occurs if some tests pass and others fail.

[orta]

It's showing a green tick for the test that successfully passed, but not for the missing result. Perhaps we need to hook in to uncaught exceptions and raise an XCTFail. Any chance you can post the crashing code or can make a PR tackling this?

[mfclarke]

The crashing code is simply [NSException raise:@"Forced Crash" format:@"Requested crash from -[CrashingObject crash]"]

I will have a crack at a PR if I have some spare time but I'm afraid that may not be for a week or two.

[blackivory86]

We're experiencing similar issues. Restarting XCode is needed to solve the issue in Test navigator pane. The Xcode test reports are showing the correct results reported in the console

[Goles]

I often clear my Derived Data directory to fix this, to be honest it's driving me crazy! haha, it seems to get screwed after a while, and then I cleanup Derived Data again.

[dbarden]

This seems to be an issue with XCTest and Xcode. What I'm seeing is that the second time the you run a spec similar to the one above, the last test does not appear on the test navigator.

Since Quick is suffering with the same problem, I guessed that the problem might not even be Specta, but XCTest.

So I did what the core of Specta does, which is to create the test on runtime by overriding +[XCTestCase testInvocations] and creating some tests at runtime.

It's something like this. It creates 3 tests a at runtime

The Test Navigator will report only 2 tests running, even though the report and the console will report 3 correctly.

What's more interesting is that the results on test navigator are correct until the method -[_XCTestDriverTestObserver testBundleDidFinish:] is hit. During the execution of that method, the test navigator is updated and one result disappears..

I tested Specta, Quick and Cedar and all of them had this behaviour, what makes the point that the Test Navigator and runtime tests are not in their best terms currently.

[modocache]

Potentially related to https://github.com/Quick/Quick/issues/373, with a proposed solution here: https://github.com/Quick/Quick/issues/439.

[oronbz]

+1, currently there is no way to run single file (even with fdescribe and fit), and when you have lots of tests it's pretty annoying running them all every time.