XCPretty not reporting failed tests, returning total test count as passed with 0 failures

[szymon-kazmierczak]

When running a build using xcodebuild and xcpretty, the failures are not being reported. XCTest/Earlgrey test suite. everything works fine when running via xcode GUI.

For example: in the ExistingRequestTests - there should be 3 tests, but only 1 is displayed in the results below.

The command used

set -o pipefail && env NSUnbufferedIO=YES xcodebuild -scheme EarlGreyTestingAppTests -project zSDK.xcodeproj -destination 'platform=iOS Simulator,id=CDC72639-4570-478A-96BF-40E0EC4353BE' -derivedDataPath 'DerivedData' -only-testing:EarlGreyTestingAppTests build-for-testing | tee '/Users/personal/Library/Logs/scan/EarlGreyTestingAppTests-EarlGreyTestingAppTests.log' | xcpretty  --report html --output '/Users/personal/code/zSDK/fastlane/test_output/earlGreyReport.html' --report junit --output '/Users/personal/code/zSDK/fastlane/test_output/earlGreyReport.xml' --report junit --output '/var/folders/kt/p4sh9kns1qq21w9vhf_j1rg00000gp/T/junit_report20180619-48080-yv54zh'

output

Test Suite EarlGreyTestingAppTests.xctest started
ExistingRequestTests
    ✓ test1 (2.682 seconds)
HCTests
    ✓ test2 (8.310 seconds)
    ✓ test3 (2.446 seconds)
    ✓ test4 (2.445 seconds)
RequestAgentTests
    ✓ test5 (2.432 seconds)
    ✓ test6 (2.502 seconds)
    ✓ test7 (2.534 seconds)
    ✓ test8 (2.471 seconds)
RequestConversationsOffTests
    ✓ test9 (4.737 seconds)
    ✓ test10 (3.621 seconds)
    ✓ test11 (5.232 seconds)
    ✓ test12 (9.138 seconds)
    ✓ test13 (2.483 seconds)
    ✓ test14 (2.293 seconds)
    ✓ test15 (8.776 seconds)
RequestListTests
    ✓ test16 (2.782 seconds)
    ✓ test17 (2.872 seconds)
    ✓ test18 (2.537 seconds)
    ✓ test19 (2.530 seconds)
    ✓ test20 (2.528 seconds)
RequestTests
    ✓ test21 (2.631 seconds)
    ✓ test22 (6.359 seconds)
    ✓ test23 (12.263 seconds)
    ✓ test24 (6.462 seconds)
    ✓ test25 (6.648 seconds)
    ✓ test26 (2.763 seconds)
    ✓ test27 (3.800 seconds)
    ✓ test28 (8.294 seconds)
    ✓ test29 (10.636 seconds)
    ✓ test30 (7.022 seconds)
    ✓ test31 (12.937 seconds)
    ✓ test32 (6.800 seconds)


	 Executed 40 tests, with 0 failures (0 unexpected) in 196.836 (196.860) seconds

▸ Test Succeeded

[modamoda]

Similar issue here. it just "hides" failed test (maybe parsing issue?)

What happened?

environment

$ xcodebuild -version
Xcode 9.2
Build version 9C40b

output with xcpretty

$xcodebuild -scheme MyAwsomeApp -configuration Debug -destination 'platform=iOS Simulator,name=iPhone 7,OS=11.2' ONLY_ACTIVE_ARCH=YES test -only-testing:MyAwsomeAppTests/MyModelFactorySpec -verbose | xcpretty

....

▸ Signing /Users/me/Library/Developer/Xcode/DerivedData/MyAwsomeApp-randomalphabetstring/Build/Products/Debug-iphonesimulator/MyAwsomeAppTests.xctest
2018-08-08 16:12:17.911 xcodebuild[52472:11447198]  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/var/folders/deep/inside/rabbit/hole/Session-MyAwsomeAppTests-2018-08-08_161217-5zdJQv.log
2018-08-08 16:12:17.911 xcodebuild[52472:11447148] [MT] IDETestOperationsObserverDebug: (RANDOM-GUID-STRING) Beginning test session MyAwsomeAppTests-RANDOM-GUID-STRING at 2018-08-08 16:12:17.911 with Xcode 9C40b on target <DVTiPhoneSimulator: 0x7faeba8ebdd0> {
		SimDevice: iPhone 7 (SIMULATOR-GUID-STRING, iOS 11.2, Booted)
} (11.2 (15C107))
** TEST FAILED **

Selected tests
Test Suite MyAwsomeAppTests.xctest started
MyModelFactorySpec
    ✓ A_Local_Controller_Factory__test01 (0.044 seconds)
    ✓ A_Local_Controller_Factory__test02 (0.047 seconds)
    ✓ A_Local_Controller_Factory__test03 (0.076 seconds)
    ✓ A_Local_Controller_Factory__test04 (0.370 seconds)
Selected tests
Test Suite MyAwsomeAppTests.xctest started
MyModelFactorySpec
    ✓ A_Local_Controller_Factory__test06 (0.092 seconds)


	 Executed 6 tests, with 1 failure (0 unexpected) in 0.092 (0.103) seconds

output without xcpretty

$ xcodebuild -scheme MyAwsomeApp -configuration Debug -destination 'platform=iOS Simulator,name=iPhone 7,OS=11.2' ONLY_ACTIVE_ARCH=YES test -only-testing:MyAwsomeAppTests/MyModelFactorySpec -verbose

...

CodeSign /Users/me/Library/Developer/Xcode/DerivedData/MyAwsomeApp-randomalphabetstring/Build/Products/Debug-iphonesimulator/MyAwsomeAppTests.xctest
    cd /workplace/me/MyAwsomeApp/src/MyAwsomeApp-sub
    export CODESIGN_ALLOCATE=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/codesign_allocate
    export PATH="/path1:/path2"

Signing Identity:     "-"

    /usr/bin/codesign --force --sign - --timestamp=none /Users/me/Library/Developer/Xcode/DerivedData/MyAwsomeApp-randomalphabetstring/Build/Products/Debug-iphonesimulator/MyAwsomeAppTests.xctest

Test Suite 'Selected tests' started at 2018-08-08 16:16:37.754
Test Suite 'MyAwsomeAppTests.xctest' started at 2018-08-08 16:16:37.755
Test Suite 'MyModelFactorySpec' started at 2018-08-08 16:16:37.755
Test Case '-[MyAwsomeAppTests.MyModelFactorySpec A_Local_Controller_Factory__test01]' started.
Test Case '-[MyAwsomeAppTests.MyModelFactorySpec A_Local_Controller_Factory__test01]' passed (0.070 seconds).
Test Case '-[MyAwsomeAppTests.MyModelFactorySpec A_Local_Controller_Factory__test02]' started.
Test Case '-[MyAwsomeAppTests.MyModelFactorySpec A_Local_Controller_Factory__test02]' passed (0.097 seconds).
Test Case '-[MyAwsomeAppTests.MyModelFactorySpec A_Local_Controller_Factory__test03]' started.
Test Case '-[MyAwsomeAppTests.MyModelFactorySpec A_Local_Controller_Factory__test03]' passed (0.131 seconds).
Test Case '-[MyAwsomeAppTests.MyModelFactorySpec A_Local_Controller_Factory__test04]' started.
Test Case '-[MyAwsomeAppTests.MyModelFactorySpec A_Local_Controller_Factory__test04]' passed (0.516 seconds).
Test Case '-[MyAwsomeAppTests.MyModelFactorySpec A_Local_Controller_Factory__test05]' started.
Could not cast value of type 'MyAwsomeApp.MyAnotherModel' (0x121616fc0) to 'MyAwsomeApp.MyModel' (0x121618e38).
2018-08-08 16:16:38.676308-0700 xctest[53504:11475955] Could not cast value of type 'MyAwsomeApp.MyAnotherModel' (0x121616fc0) to 'MyAwsomeApp.MyModel' (0x121618e38).
2018-08-08 16:16:41.218 xcodebuild[53272:11475693]  IDETestOperationsObserverDebug: Writing diagnostic log for test session to:
/var/folders/rabbit/hole/Session-MyAwsomeAppTests-2018-08-08_161641-HzUOdX.log
2018-08-08 16:16:41.219 xcodebuild[53272:11474706] [MT] IDETestOperationsObserverDebug: (RANDOM-GUID-STRING) Beginning test session MyAwsomeAppTests-RANDOM-GUID-STRING at 2018-08-08 16:16:41.219 with Xcode 9C40b on target <DVTiPhoneSimulator: 0x7fce5ffe2900> {
		SimDevice: iPhone 7 (SIMULATOR-GUID-STRING, iOS 11.2, Booted)
} (11.2 (15C107))

Restarting after unexpected exit or crash in MyModelFactorySpec/A_Local_Controller_Factory__test05(); summary will include totals from previous launches.

Test Suite 'Selected tests' started at 2018-08-08 16:16:42.105
Test Suite 'MyAwsomeAppTests.xctest' started at 2018-08-08 16:16:42.106
Test Suite 'MyModelFactorySpec' started at 2018-08-08 16:16:42.106
Test Case '-[MyAwsomeAppTests.MyModelFactorySpec A_Local_Controller_Factory__test06]' started.
Test Case '-[MyAwsomeAppTests.MyModelFactorySpec A_Local_Controller_Factory__test06]' passed (0.050 seconds).
Test Suite 'MyModelFactorySpec' failed at 2018-08-08 16:16:42.157.
	 Executed 6 tests, with 1 failure (0 unexpected) in 0.050 (0.051) seconds
Test Suite 'MyAwsomeAppTests.xctest' failed at 2018-08-08 16:16:42.158.
	 Executed 6 tests, with 1 failure (0 unexpected) in 0.050 (0.052) seconds
Test Suite 'Selected tests' failed at 2018-08-08 16:16:42.158.
	 Executed 6 tests, with 1 failure (0 unexpected) in 0.050 (0.053) seconds
** TEST FAILED **

What is issue?

in the output without xcpretty, we can see A_Local_Controller_Factory__test05 is failed due to invalid type casting. so it's failed.

While we're expecting error message output like this:

    ✗ A_Local_Controller_Factory__test05, fail because blah blah ....

it does not give us any information - which test is failed and why. (--report does not show failed test case either)