Bluepill should ignore test bundles without hosts

[loufranco]

My workspace just added a very simple swift module with tests -- they don't need a host to run (i.e "Host Application" is set to None in Xcode and the "TestHostBundleIdentifier" isn't present in the dictionary we load from the xctest file)

I already tested a change locally (that works) where I skipped test files that don't have the TestHostBundleIdentifier key.

I can PR if this seems like the right approach.

It means that Bluepill won't run these tests -- but at least you could still run the others -- right now it just give an error and stops.

[oliverhu]

That sounds weird. We used to have No Host Application tests, and we run them with Bluepill. I think you need this flag:

additional-unit-xctests | n/a | Additional XCTest bundles that is not Plugin folder | N | n/a
-- | -- | -- | -- | --

[loufranco]

I don't want to run them -- I just want to run a GUI test in another test bundle.

But when it tries to load the entire .xctest file, it can't load this one. It looks to me that there's partially another bug stopping it, but it also looks for "TestHostBundleIdentifier" in the dict (and it's not there), so it returns nil and the caller says that nil is an error.

In my case, I have a workspace like this

workspace -> Module (very simple -- just Swift structs and funcs -- no UIKit or app deps) -> App that uses Module

Module has a test target in it that doesn't need a Host App has Unit and UI tests that both use the app as host

When I try to just run 1 GUI test in the GUI test target of the app, it tries to load all of the test bundles in xctest and it can't -- it doesn't process the TESTROOT in the paths correctly -- perhaps that's the real issue? I'll try to make a simple test case.

[oliverhu]

Got it, that is possible. We only validated it works for our specific case. It would be great if you could provide a sample project to reproduce this, feel free to submit a PR (draft is fine) to help us understand the issue.

[loufranco]

Ok, here's a minimal workspace that reproduces the issue. AppWithModuleTests.zip

I included the .xctestrun file in the zip. When I run bluepill against it, I get

[  ERROR ] Failed to read data for SimpleModuleTests

The first problem I see is that bluepill doesn't correctly substitute __TESTROOT__ in the bundle, but when I tried to fix that, then it wanted the app bundle id (which doesn't exist).

I need some guidance for how to fix this -- I am willing to make a PR.

[oliverhu]

Sorry for getting back to you late. What command did you use to run against that demo? Like this?

bluepill --xctestrun-path path_to_xcruntest

We should include those test bundles without test hosts. I did find there are some potential issues in replacing PLATFORMS & TESTHOSTS & ... , looks should not be a if / else if / else if if but if if if

[oliverhu]

Figured out how to run it.. the error I got is:

Domain=FBSOpenApplicationServiceErrorDomain Code=1 "The request to open "com.loufranco.SimpleModuleTests" failed." UserInfo={BSErrorCodeDescription=RequestDenied, NSUnderlyingError=0x1023ab6d0 {Error Domain=FBSOpenApplicationErrorDomain Code=1 "(null)" UserInfo={BSErrorCodeDescription=Unspecified}}, NSLocalizedDescription=The request to open "com.loufranco.SimpleModuleTests" failed., FBSOpenApplicationRequestID=65250, NSLocalizedFailureReason=The request was denied by service delegate (SBMainWorkspace) for reason: Unspecified.} - -1

The issue seems to be we are trying to launch the module but apparently we can only launch application instead of a module.

[oliverhu]

Now I remember how we resolved the problem, we created a dummy app as the host for the tests that don't need a test host.

[oliverhu]

This is my local diff btw:

index e78049a..d88d8fc 100644
--- a/Source/Shared/BPXCTestFile.m
+++ b/Source/Shared/BPXCTestFile.m
@@ -113,7 +113,6 @@ NSString *objcNmCmdline = @"nm -U '%@' | grep ' t ' | cut -d' ' -f3,4 | cut -d'-
     NSString * const TESTHOST = @"__TESTHOST__";
     NSString * const PLATFORMS = @"__PLATFORMS__";
     NSString *testHostPath = [dict objectForKey:@"TestHostPath"];
-    NSString *testHostBundleIdentifier = [dict objectForKey:@"TestHostBundleIdentifier"];
     if (!testHostPath) {
         BP_SET_ERROR(error, @"No 'TestHostPath' found");
         return nil;
@@ -129,13 +128,14 @@ NSString *objcNmCmdline = @"nm -U '%@' | grep ' t ' | cut -d' ' -f3,4 | cut -d'-
      */
     if ([testBundlePath rangeOfString:TESTHOST].location != NSNotFound) {
         testBundlePath = [testBundlePath stringByReplacingOccurrencesOfString:TESTHOST withString:testHostPath];
-    } else if ([testBundlePath rangeOfString:TESTROOT].location != NSNotFound) {
+    }
+    if ([testBundlePath rangeOfString:TESTROOT].location != NSNotFound) {
         testBundlePath = [testBundlePath stringByReplacingOccurrencesOfString:TESTROOT withString:testRoot];
-    } else if ([testHostPath rangeOfString:PLATFORMS].location != NSNotFound) {
+    }
+    if ([testHostPath rangeOfString:PLATFORMS].location != NSNotFound) {
         NSString *platformsPath = [xcodePath stringByAppendingPathComponent:@"Platforms"];
+        testHostPath = [testBundlePath stringByReplacingOccurrencesOfString:PLATFORMS withString:testHostPath];
         testBundlePath = [testBundlePath stringByReplacingOccurrencesOfString:PLATFORMS withString:platformsPath];
-    } else {
-        [BPUtils printInfo:ERROR withString:@"testBundlePath is incorrect, please check xctestrun file"];
     }
     NSString * UITargetAppPath = [dict objectForKey:@"UITargetAppPath"];
     if (UITargetAppPath) {```

[loufranco]

Thanks for looking into this -- I will try this out locally on my real project and report back

[oliverhu]

I also remembered why it worked out even before we set up the dummy app. That was because we didn't use the xctestrun file to extract the information to launch the application & run test - we ask user to specify the location of the app, so we launch it anyway even though an application launch / simulator is not needed.

I am actually leaning towards skipping the test bundles without hosts and throw a warning for now. I don't think a simulator is needed at all to run the unit tests without hosts. An ideal solution would be refactoring

- (void)launchApplicationAndExecuteTestsWithParser:(BPTreeParser *)parser andCompletion:(void (^)(NSError *, pid_t))completion;

method to another version that accept executeTests without launching the application (if there is no application host specified), you could also give this a try.

Feel free to submit a PR, also need to update READ.ME to include this information :)

[chenxiao0228]

@loufranco did the workaround work for you?