swiftlang
Compiling many Swift Testing tests is significantly slower than XCTest tests
Description
Compiling a large amount of Swift Testing tests is significantly slower than compiling the same amount of tests using XCTest on a clean build.
At 1000 tests in a single file building a test suite with Swift Testing is 24x slower than an almost identical suite using XCTest.
Here's a repo with a project to benchmark: https://github.com/marcosgriselli/swift-testing-performance. A few results from that project:
1000 Tests in a single file:
➜ swift-testing-performance git:(main) ✗ ./benchmark.py --test_count 1000 --runs 5
Benchmark 1: XCTest (1000 tests)
Time (mean ± σ): 3.059 s ± 0.090 s [User: 0.658 s, System: 0.214 s]
Range (min … max): 2.965 s … 3.183 s 5 runs
Benchmark 2: SwiftTesting (1000 tests)
Time (mean ± σ): 73.940 s ± 0.341 s [User: 0.747 s, System: 0.285 s]
Range (min … max): 73.603 s … 74.427 s 5 runs
Summary
XCTest (1000 tests) ran
24.17 ± 0.72 times faster than SwiftTesting (1000 tests)
Performance significantly improves when splitting the tests into different files, still 10x slower than XCTest.
1000 tests split in 10 files:
➜ swift-testing-performance git:(main) ./benchmark.py --test_count 1000 --test_files 10
Benchmark 1: XCTest (1000 tests in 10 files)
Time (mean ± σ): 2.010 s ± 0.166 s [User: 0.665 s, System: 0.216 s]
Range (min … max): 1.855 s … 2.185 s 3 runs
Benchmark 2: SwiftTesting (1000 tests in 10 files)
Time (mean ± σ): 20.769 s ± 0.351 s [User: 0.792 s, System: 0.295 s]
Range (min … max): 20.531 s … 21.171 s 3 runs
Summary
XCTest (1000 tests in 10 files) ran
10.33 ± 0.87 times faster than SwiftTesting (1000 tests in 10 files)
I'm aware of the macros compilation build time issues outside of swift-syntax and that the team is working on addressing these, is there anything we can do in the meantime to reduce the impact while adopting Swift Testing?
Expected behavior
No response
Actual behavior
No response
Steps to reproduce
Run benchmarks on the shared repo to measure performance.
./benchmarks.py --test_count 1000 --test_files 1 --runs 5
./benchmarks.py --test_count 1000 --test_files 10 --runs 5
swift-testing version/commit hash
Xcode 16.0 (16A5171c)
Swift & OS version (output of swift --version ; uname -a)
swift-driver version: 1.109.2 Apple Swift version 6.0 (swiftlang-6.0.0.3.300 clang-1600.0.20.10) Target: arm64-apple-macosx14.0 Darwin Marcoss-MBP.attlocal.net 23.5.0 Darwin Kernel Version 23.5.0: Wed May 1 20:12:58 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6000 arm64
Thank you for this report @marcosgriselli! The example project and data are very helpful. I am investigating this along with @rintaro.
Does this need to move over to swift-syntax or swift?
At this point I do believe it should be transferred to the swiftlang/swift repo. I just tried doing that though, and it appears that you cannot transfer issues across organization boundaries, and due to the pending apple → swiftlang organization transition we'll need to wait until swift-testing makes the transition. We expect this to happen before too long.
@stmontgomery happy to close and create the issue there unless you want the transfer to keep the issue history clean
We can wait to transfer. The relevant people know about the issue already.
I'm actively looking into this issue:
- https://github.com/swiftlang/swift/pull/75115
- https://github.com/swiftlang/swift/pull/75130
So far, these patches improves the swift-testing test compilation performance by 2.6x or so (tested with a file with 500 @Test/#expect cases) . I'm exploring other areas for further improvements.
@rintaro Does this seem more like a compiler or swift-syntax issue, to you? Is there something we ought to be doing in Swift Testing at this stage?
Just to share another data point, moving from XCTest to Swift Testing increased our CI test job from 14 minutes to 25 (Xcode 16.2).
Is this still being looked into?
Just to share another data point, moving from XCTest to Swift Testing increased our CI test job from 14 minutes to 25 (Xcode 16.2).
@NachoSoto Is that measurement including the time taken to run the tests? This issue is focused on build times only. Of course, there still may be a separate runtime issue we need to analyze there, but I just want to clarify. If you are including runtime, do you have parallelization enabled in one or both of XCTest or Swift Testing?
It includes the entire job, so yes including runtime both before and after.
We were fully serial before, but we did enable parallelization for a lot of the Swift Testing suites. So if anything runtime should have gotten faster.
It may be helpful to collect the build and runtime measurements separately to break down the time differences more granularly. You can use the xcodebuild build-for-testing and xcodebuild test-without-building commands, respectively.
If using swift test, consider swift build --build-tests and swift test --skip-build too.
Another data point to clarify: are you building Swift Testing as a package dependency (from source) or are you using the copy included in the Swift 6.x toolchain?
Sorry for the false alarm, this turned out to be a GitHub issue that happened right as we merged our migration: https://github.com/actions/runner-images/issues/11827#issuecomment-2734028720
2025 update for anyone finding this thread. I benchmarked again using Xcode 16.3 beta 3 (16E5129f) with the following results:
1 file - 1000 tests:
➜ swift-testing-performance git:(main) ./benchmark.py --test_count 1000 --runs 3
Benchmark 1: XCTest (1000 tests in 1 files)
Time (mean ± σ): 3.217 s ± 0.422 s [User: 0.394 s, System: 0.176 s]
Range (min … max): 2.940 s … 3.702 s 3 runs
Benchmark 2: SwiftTesting (1000 tests in 1 files)
Time (mean ± σ): 20.254 s ± 0.372 s [User: 0.391 s, System: 0.171 s]
Range (min … max): 19.972 s … 20.675 s 3 runs
Summary
XCTest (1000 tests in 1 files) ran
6.30 ± 0.83 times faster than SwiftTesting (1000 tests in 1 files)
100 tests per file:
➜ swift-testing-performance git:(main) ./benchmark.py --test_count 1000 --runs 3 --test_files 10
Benchmark 1: XCTest (1000 tests in 10 files)
Time (mean ± σ): 3.285 s ± 0.253 s [User: 0.409 s, System: 0.194 s]
Range (min … max): 3.137 s … 3.578 s 3 runs
Warning: The first benchmarking run for this command was significantly slower than the rest (3.578 s). This could be caused by (filesystem) caches that were not filled until after the first run. You are already using the '--prepare' option which can be used to clear caches. If you did not use a cache-clearing command with '--prepare', you can either try that or consider using the '--warmup' option to fill those caches before the actual benchmark.
Benchmark 2: SwiftTesting (1000 tests in 10 files)
Time (mean ± σ): 5.107 s ± 0.009 s [User: 0.410 s, System: 0.182 s]
Range (min … max): 5.098 s … 5.116 s 3 runs
Summary
XCTest (1000 tests in 10 files) ran
1.55 ± 0.12 times faster than SwiftTesting (1000 tests in 10 files)
Even though the single file approach is still significantly slower the second test matches real scenarios more closely and 1.55x slower is now reasonable in comparison to where this issue started given all the Macros expansion happening. This time hit will probably be recovered with other features Swift Testing offers.
I'd vote in favor of closing this issue as resolved since the original reason doesn't reflect the current state and can mislead people that don't read the full updates.
Thanks for those statistics! I agree it sounds like we can close this as resolved at this point. The timing between the two libraries will never be identical, but 1.55x is a reasonable baseline and is something we can iteratively improve over time.