rules_xcodeproj icon indicating copy to clipboard operation
rules_xcodeproj copied to clipboard

Published 20 hours ago • verified publisher icon MobileNativeFoundation

Enable integration test with Bazel 6.0.0-pre.XXXX

Open cgrindel opened this issue 2 years ago • 2 comments

The specs for the ios_app example differ when generated using Bazel 5.0.0 and 6.0.0-pre.20220223.1. See below for an example. To enable testing with a different version of Bazel, we can do one of the following:

  1. Only perform the spec test on the "current" version of Bazel. The other Bazel versions will skip the spec test.
  2. Perform the spec test against the checked-in specs using the current Bazel version. For the other Bazel versions, we would execute the fixture update command before running the spec test. The benefit of this approach is that it exercises the spec testing code on different Bazel versions.
  3. Transition away from spec tests and write tests that assert the existence of certain elements in the output.

Example Diff Comparing Spec Generated using Bazel 5.0.0 and Bazel 6.0.0-pre.20220223.1

==================== Test output for //test/fixtures:xcodeproj_fixtures:
Spec doesn't match expected:
39c39
<         "//Example:Example.library ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0",
---
>         "//Example:Example.library ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd",
41c41
<             "//Example:Example applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0"
---
>             "//Example:Example applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd"
43c43
<         "//ExampleTests:ExampleTests.library ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0",
---
>         "//ExampleTests:ExampleTests.library ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd",
45c45
<             "//ExampleTests:ExampleTests.__internal__.__test_bundle applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0"
---
>             "//ExampleTests:ExampleTests.__internal__.__test_bundle applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd"
47c47
<         "//ExampleUITests:ExampleUITests.library ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0",
---
>         "//ExampleUITests:ExampleUITests.library ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd",
49c49
<             "//ExampleUITests:ExampleUITests.__internal__.__test_bundle applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0"
---
>             "//ExampleUITests:ExampleUITests.__internal__.__test_bundle applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd"
53,54c53,54
<         "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/TestingUtils/libTestingUtils.a",
<         "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/Utils/libUtils.a"
---
>         "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/TestingUtils/libTestingUtils.a",
>         "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/Utils/libUtils.a"
57c57
<         "//Example:Example applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0",
---
>         "//Example:Example applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd",
108c108
<             "configuration": "applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0",
---
>             "configuration": "applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd",
110c110
<                 "//Example:Example.library ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0"
---
>                 "//Example:Example.library ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd"
115,116c115,116
<                 "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/Utils/libUtils.a",
<                 "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/Example/libExample.library.a"
---
>                 "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/Utils/libUtils.a",
>                 "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/Example/libExample.library.a"
131c131
<                 "path": "bazel-out/applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/Example/Example_archive-root/Payload/Example.app",
---
>                 "path": "bazel-out/applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/Example/Example_archive-root/Payload/Example.app",
137c137
<         "//Example:Example.library ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0",
---
>         "//Example:Example.library ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd",
186c186
<             "configuration": "ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0",
---
>             "configuration": "ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd",
188c188
<                 "//Utils:Utils ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0"
---
>                 "//Utils:Utils ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd"
200c200
<                     "_": "ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/Utils/Utils.swift.modulemap",
---
>                     "_": "ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/Utils/Utils.swift.modulemap",
208,210c208,210
<                     "swiftdoc": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/Example/Example.swiftdoc",
<                     "swiftmodule": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/Example/Example.swiftmodule",
<                     "swiftsourceinfo": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/Example/Example.swiftsourceinfo"
---
>                     "swiftdoc": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/Example/Example.swiftdoc",
>                     "swiftmodule": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/Example/Example.swiftmodule",
>                     "swiftsourceinfo": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/Example/Example.swiftsourceinfo"
221c221
<                 "path": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/Example/libExample.library.a",
---
>                 "path": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/Example/libExample.library.a",
227c227
<         "//ExampleTests:ExampleTests.__internal__.__test_bundle applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0",
---
>         "//ExampleTests:ExampleTests.__internal__.__test_bundle applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd",
278c278
<             "configuration": "applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0",
---
>             "configuration": "applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd",
280,281c280,281
<                 "//ExampleTests:ExampleTests.library ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0",
<                 "//Example:Example applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0"
---
>                 "//ExampleTests:ExampleTests.library ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd",
>                 "//Example:Example applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd"
286,287c286,287
<                 "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/TestingUtils/libTestingUtils.a",
<                 "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/ExampleTests/libExampleTests.library.a"
---
>                 "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/TestingUtils/libTestingUtils.a",
>                 "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/ExampleTests/libExampleTests.library.a"
302c302
<                 "path": "bazel-out/applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/ExampleTests/ExampleTests.__internal__.__test_bundle_archive-root/ExampleTests.xctest",
---
>                 "path": "bazel-out/applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/ExampleTests/ExampleTests.__internal__.__test_bundle_archive-root/ExampleTests.xctest",
306c306
<             "test_host": "//Example:Example applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0"
---
>             "test_host": "//Example:Example applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd"
308c308
<         "//ExampleTests:ExampleTests.library ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0",
---
>         "//ExampleTests:ExampleTests.library ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd",
358c358
<             "configuration": "ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0",
---
>             "configuration": "ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd",
360,362c360,362
<                 "//Example:Example.library ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0",
<                 "//TestingUtils:TestingUtils ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0",
<                 "//Utils:Utils ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0"
---
>                 "//Example:Example.library ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd",
>                 "//TestingUtils:TestingUtils ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd",
>                 "//Utils:Utils ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd"
373c373
<                     "_": "ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/Utils/Utils.swift.modulemap",
---
>                     "_": "ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/Utils/Utils.swift.modulemap",
381,383c381,383
<                     "swiftdoc": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/ExampleTests/ExampleTests.swiftdoc",
<                     "swiftmodule": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/ExampleTests/ExampleTests.swiftmodule",
<                     "swiftsourceinfo": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/ExampleTests/ExampleTests.swiftsourceinfo"
---
>                     "swiftdoc": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/ExampleTests/ExampleTests.swiftdoc",
>                     "swiftmodule": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/ExampleTests/ExampleTests.swiftmodule",
>                     "swiftsourceinfo": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/ExampleTests/ExampleTests.swiftsourceinfo"
394c394
<                 "path": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/ExampleTests/libExampleTests.library.a",
---
>                 "path": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/ExampleTests/libExampleTests.library.a",
400c400
<         "//ExampleUITests:ExampleUITests.__internal__.__test_bundle applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0",
---
>         "//ExampleUITests:ExampleUITests.__internal__.__test_bundle applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd",
451c451
<             "configuration": "applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0",
---
>             "configuration": "applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd",
453,454c453,454
<                 "//ExampleUITests:ExampleUITests.library ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0",
<                 "//Example:Example applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0"
---
>                 "//ExampleUITests:ExampleUITests.library ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd",
>                 "//Example:Example applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd"
459c459
<                 "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/ExampleUITests/libExampleUITests.library.a"
---
>                 "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/ExampleUITests/libExampleUITests.library.a"
474c474
<                 "path": "bazel-out/applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/ExampleUITests/ExampleUITests.__internal__.__test_bundle_archive-root/ExampleUITests.xctest",
---
>                 "path": "bazel-out/applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/ExampleUITests/ExampleUITests.__internal__.__test_bundle_archive-root/ExampleUITests.xctest",
478c478
<             "test_host": "//Example:Example applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0"
---
>             "test_host": "//Example:Example applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd"
480c480
<         "//ExampleUITests:ExampleUITests.library ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0",
---
>         "//ExampleUITests:ExampleUITests.library ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd",
530c530
<             "configuration": "ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0",
---
>             "configuration": "ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd",
545,547c545,547
<                     "swiftdoc": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/ExampleUITests/ExampleUITests.swiftdoc",
<                     "swiftmodule": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/ExampleUITests/ExampleUITests.swiftmodule",
<                     "swiftsourceinfo": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/ExampleUITests/ExampleUITests.swiftsourceinfo"
---
>                     "swiftdoc": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/ExampleUITests/ExampleUITests.swiftdoc",
>                     "swiftmodule": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/ExampleUITests/ExampleUITests.swiftmodule",
>                     "swiftsourceinfo": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/ExampleUITests/ExampleUITests.swiftsourceinfo"
558c558
<                 "path": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/ExampleUITests/libExampleUITests.library.a",
---
>                 "path": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/ExampleUITests/libExampleUITests.library.a",
564c564
<         "//TestingUtils:TestingUtils ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0",
---
>         "//TestingUtils:TestingUtils ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd",
614c614
<             "configuration": "ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0",
---
>             "configuration": "ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd",
619c619
<                         "_": "ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/TestingUtils/TestingUtils.swift",
---
>                         "_": "ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/TestingUtils/TestingUtils.swift",
631,633c631,633
<                     "swiftdoc": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/TestingUtils/TestingUtils.swiftdoc",
<                     "swiftmodule": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/TestingUtils/TestingUtils.swiftmodule",
<                     "swiftsourceinfo": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/TestingUtils/TestingUtils.swiftsourceinfo"
---
>                     "swiftdoc": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/TestingUtils/TestingUtils.swiftdoc",
>                     "swiftmodule": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/TestingUtils/TestingUtils.swiftmodule",
>                     "swiftsourceinfo": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/TestingUtils/TestingUtils.swiftsourceinfo"
644c644
<                 "path": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/TestingUtils/libTestingUtils.a",
---
>                 "path": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/TestingUtils/libTestingUtils.a",
650c650
<         "//Utils:Utils ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0",
---
>         "//Utils:Utils ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd",
698c698
<             "configuration": "ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0",
---
>             "configuration": "ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd",
722c722
<                 "path": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin/Utils/libUtils.a",
---
>                 "path": "bazel-out/ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin/Utils/libUtils.a",
730c730
<                         "_": "ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-4c6d2b462eb0/bin",
---
>                         "_": "ios-x86_64-min15.0-applebin_ios-ios_x86_64-fastbuild-ST-d3e756bfe7fd/bin",

Run `bazel run //test/fixtures:update` if you wish to accept these changes.
================================================================================

cgrindel avatar Mar 19 '22 17:03 cgrindel

There is value when I'm developing to be able to see how the spec changes and the project. Some of it is implementation detail though.

I plan to address this first by creating tests that ensure that the project can be built with Xcode. After that I'll need to find a way to verify that most behavior works (which can be tricky for things like SwiftUI previews).

After those are in place, I can change the fixture tests to be smarter in how they compare.

brentleyjones avatar Mar 21 '22 16:03 brentleyjones

Thoughts from @brentleyjones regarding Xcode testing and possibly migrating away from fixture tests.

Something that would help prevent regressions: a test that runs xcodebuild build on the fixture projects and ensure that they can build. And then beyond that a similar one to make sure the tests (via xcodebuld test) run (which, with the UI tests, will make sure that the app can launch without crashing).

cgrindel avatar Apr 11 '22 19:04 cgrindel