rules_apple UI tests don't work with --test_filter unless ProductModuleName matches source lib

did bazel support running testing with spesific case only?

I know xcodebuild has this option called -only-testing, which can be used by passing an array of Test Bundle/Test Suite/Test Cases. Screen Shot 2022-02-17 at 15 32 52

do bazel have something equivalent with that?

Feb 17 '22 08:02 wendyliga

Yes via --test_filter but I believe it's up to your test runner to use it correctly.

--test_filter=TestClass/testName should work with xctestrunner

Feb 17 '22 15:02 jszumski

Please submit an issue if this doesn't work!

Feb 17 '22 16:02 keith

after looking to --test_filter, I find it only working for ios_unit_test and didn't work for ios_ui_test

from this example repo

bazel test ios/DummyTest --test_filter=ios_DummyTest_Source.DummyTests/test

this will run DummyTest and filter only DummyTests/test Screen Shot 2022-02-23 at 10 41 46

but

bazel test ios/DummyUITest --test_filter=ios_DummyUITest_Source.UITest/test

Screen Shot 2022-02-23 at 10 40 55

do you guys have any idea? I try multiple different combinations of class bundle names and function, but none work.

Feb 23 '22 03:02 wendyliga

Can you show the output when your test runs so we can see the names of things to verify that matches? it's possible this isn't implemented correctly / at all for UI tests, I'm not sure

Feb 24 '22 19:02 keith

@keith this is the log, running full test Screen Shot 2022-02-25 at 03 37 15

Feb 24 '22 20:02 wendyliga

Ah I think you need to update rules_apple / xctestrunner to include this fix https://github.com/google/xctestrunner/commit/24ba1517795e6e1c99d361988b95e51a48489087

Feb 28 '22 20:02 keith

If your target names have dashes in them you also need https://github.com/google/xctestrunner/pull/42

Feb 28 '22 20:02 keith

I solved the issue by using this naming conversion

{bundle_name}/{class_name}/{function_name}

bazel test ios/DummyUITest --test_filter=ios_DummyUITest_Source.UITest/test

working

bazel test ios/DummyUITest --test_filter=ios_DummyUITest_Source/UITest/test

Mar 01 '22 09:03 wendyliga

but i got crash after that

Command line invocation:
    /Applications/Xcode_13.2.1.app/Contents/Developer/usr/bin/xcodebuild test-without-building -xctestrun /var/folders/tg/5hc1y5wn5qg4gnfhktnyt_kc0000gn/T/test_runner_work_dir.X5J7V7/TEST_ROOT/xctestrun.plist -destination id=874334EE-E8C9-4322-9CAC-F088B66C54DF -derivedDataPath /private/var/tmp/_bazel_wendy.liga/8284c4a94ccbbb579b9e33d50d39273c/execroot/__main__/bazel-out/darwin_arm64-fastbuild/testlogs/ios/DummyUITest/DummyUITest/test.outputs -resultBundlePath /private/var/tmp/_bazel_wendy.liga/8284c4a94ccbbb579b9e33d50d39273c/execroot/__main__/bazel-out/darwin_arm64-fastbuild/testlogs/ios/DummyUITest/DummyUITest/test.outputs/test.xcresult

User defaults from command line:
    IDEBuildOperationResultBundlePath = /private/var/tmp/_bazel_wendy.liga/8284c4a94ccbbb579b9e33d50d39273c/execroot/__main__/bazel-out/darwin_arm64-fastbuild/testlogs/ios/DummyUITest/DummyUITest/test.outputs/test.xcresult
    IDEDerivedDataPathOverride = /var/tmp/_bazel_wendy.liga/8284c4a94ccbbb579b9e33d50d39273c/execroot/__main__/bazel-out/darwin_arm64-fastbuild/testlogs/ios/DummyUITest/DummyUITest/test.outputs
    IDEPackageSupportUseBuiltinSCM = YES
    XCTHTestRunSpecificationPath = /var/folders/tg/5hc1y5wn5qg4gnfhktnyt_kc0000gn/T/test_runner_work_dir.X5J7V7/TEST_ROOT/xctestrun.plist

** INTERNAL ERROR: Uncaught exception **
Uncaught Exception: *** -[__NSSetM addObject:]: object cannot be nil
Stack:
  0   __exceptionPreprocess (in CoreFoundation)
  1   objc_exception_throw (in libobjc.A.dylib)
  2   -[__NSCFString characterAtIndex:].cold.1 (in CoreFoundation)
  3   -[__NSSetM addObject:].cold.2 (in CoreFoundation)
  4   -[__NSSetM addObject:] (in CoreFoundation)
  5   -[XCTTestIdentifierSetBuilder addTestIdentifier:] (in XCTestCore)
  6   -[XCTHTestRunSpecification initWithDictionaryRepresentation:error:] (in XCTHarness)
  7   -[XCTHTestRunCoder_V1 testRunSpecificationsInDictionaryRepresentation:removingPathPlaceholders:error:] (in XCTHarness)
  8   -[XCTHTestRunCoder_V1 testRunFileWithDictionaryRepresentation:removingPathPlaceholders:error:] (in XCTHarness)
  9   +[XCTHTestRunFileSerialization testRunFileWithDictionaryRepresentation:removingPathPlaceholders:error:] (in XCTHarness)
 10   +[XCTHTestRunFileSerialization testRunFileWithContentsOfURL:pathPlaceholders:error:] (in XCTHarness)
 11   specialized static XCTHTestRunFileSerialization.testRunFile(withContentsOf:derivedDataPath:) (in IDEFoundation)
 12   @objc static XCTHTestRunFileSerialization.testRunFile(withContentsOf:derivedDataPath:) (in IDEFoundation)
 13   -[Xcode3CommandLineBuildTool _resolveInputOptionsWithTimingSection:] (in Xcode3Core)
 14   -[Xcode3CommandLineBuildTool run] (in Xcode3Core)
 15   XcodeBuildMain (in libxcodebuildLoader.dylib)
 16   start (in dyld)

error: Result bundle at path /private/var/tmp/_bazel_wendy.liga/8284c4a94ccbbb579b9e33d50d39273c/execroot/__main__/bazel-out/darwin_arm64-fastbuild/testlogs/ios/DummyUITest/DummyUITest/test.outputs/test.xcresult does not exist.
2022-03-01 14:16:11,918 b''
2022-03-01 14:16:11,919 Deleting simulator 874334EE-E8C9-4322-9CAC-F088B66C54DF asynchronously.
2022-03-01 14:16:11,934 Done.

Mar 01 '22 14:03 wendyliga

i debugged to https://github.com/bazelbuild/rules_apple/pull/1369

Mar 01 '22 23:03 keith

@keith thanks for the response, so far to make it work

use the same name for uitest target and swift module_name.
filtering will work with --test_filter={class_name}/{function_name}, we do not need bundle_name anymore

Mar 02 '22 03:03 wendyliga

I wonder if using the bundle name == the module name would work as well

Mar 02 '22 03:03 keith

crash --test_filter={bundle_name}/{class_name}/{function_name}

works --test_filter={bundle_name}.{class_name}/{function_name}

Mar 02 '22 04:03 wendyliga

Yea I don't think the double slash variation is valid

Mar 02 '22 04:03 keith

Note we just pass these through to xcodebuild so it's up to it to validate

Mar 02 '22 04:03 keith