swift-testing icon indicating copy to clipboard operation
swift-testing copied to clipboard
verified publisher icon swiftlang

TestingMacros plugin location is inconsistent between platforms

Open ADKaster opened this issue 11 months ago • 6 comments

Motivation

Including swift-testing in the 6.x toolchains has made it easier than ever to include the testing framework in out-of-toolchain user projects.

Unfortunately, for CMake projects, some manual elbow grease is required. I'll link a swiftlang/swift issue here later that goes into more details on the hoops needed to set up a CMake project from 0.

In order to link a test main.swift against swift-testing on Linux, both the libTesting.so dylib and the libTestingMacros.so plugin dylib must be found. In order to actually find these dylibs from a Tarball or swiftly install of swift 6.0.3, there's a few steps:

  1. query swiftc -print-target-info to find the runtimeLibraryImportPaths (likely using CMake's builtin json parser/walker methods instead of jq)
$ swiftc -print-target-info | jq '.paths.runtimeLibraryImportPaths'
[
  "/home/andrew/.local/share/swiftly/toolchains/main-snapshot-2025-03-17/usr/lib/swift/linux",
  "/home/andrew/.local/share/swiftly/toolchains/main-snapshot-2025-03-17/usr/lib/swift/linux/x86_64"
]
  1. find_library for Testing, using those paths
  2. find_library for TestingMacros, using a relative path of ../host/plugins/testing from the discovered paths. There is no information from print-target-info that gives the host path, or the host plugin path.
  3. Create an imported target for SwiftTesting::SwiftTesting (or whatever one's preferred naming is).
  4. Attach some INTERFACE_COMPILE_OPTIONS to the target to either -load-plugin-library or -plugin-path to where libTestingMacros.so is.

This logic is made slightly more complicated because the path to the plugin library is different depending on platform. For a project that wants to be cross-platform, some extra legwork is required to make it worth with both Darwin and non-Darwin host toolchains.

In TestingMacros/CMakeLists.txt, there is an explicit choice: On Darwin, use lib/swift/host/plugins/testing. on non-Darwin, use lib/swift/host/plugins.

https://github.com/swiftlang/swift-testing/blob/7ccbd6884d75d8a685055dd2c15c881ea30cfd64/Sources/TestingMacros/CMakeLists.txt#L67-L71

Proposed solution

It would be less work for projects writing their own FindSwiftTesting.cmake modules if the path to libTestingMacros.{dll,dylib,so} was consistent across platforms.

Standardizing on lib/swift/host/plugins/testing would be the obvious choice, as I've been told by @stmontgomery that the symlink from lib/swift/host/plugins/libTestingMacros.dylib to lib/swift/host/plugins/testing/libTestingMacros.dylib in Xcode.app 16.2 (and probably earlier as well?) should not be relied upon.

Alternatives considered

Standardizing on host/plugins rather than host/plugins/testing could work, but presumably there's a reason to use the testing subdirectory.

Additional information

No response

ADKaster avatar Mar 27 '25 03:03 ADKaster

It would be less work for projects writing their own FindSwiftTesting.cmake modules if the path to libTestingMacros.{dll,dylib,so} was consistent across platforms.

Can you elaborate on why you need to do this? :)

grynspan avatar Mar 27 '25 15:03 grynspan

Can you elaborate on why you need to do this? :)

When working solely with Xcode.app (or probably CommandLineTools too) toolchains, a CMake project can get away with this:

find_library(SWIFT_TESTING NAMES Testing)
if (NOT SWIFT_TESTING)
   message(FATAL_ERROR "no swift testing :(")
endif()

add_executable(MySwiftTest
   TestFileA.swift
   TestFileB.swift
   TestMain.swift)

target_link_libraries(MySwiftTest ${SWIFT_TESTING})

add_test(MySwiftTest COMMAND MySwiftTest)

This will end up finding the Testing.framework in /Applications/Xcode.app/Contents/Developer/Platform/MacOSX.platform/Developer/Library/Frameworks. It also properly finds the TestingMacros plugin when building the swift files that use things like @Test or @Suite.

When working with a swift.org toolchain as either a .tar.gz or swiftly-installed version, no matter that platform, a simple find_library(Blah NAMES Testing) is not sufficient.

First, the PATHS argument needs passed with possible locations for where to find libTesting.{so,dylib}. In a Linux toolchain, this is going to be swiftc -print-target-info | jq -r ".paths.runtimeLibraryImportPaths[0]" .

  "paths": {
    "runtimeLibraryPaths": [
      "/home/andrew/.local/share/swiftly/toolchains/6.0.3/usr/lib/swift/linux"
    ],
    "runtimeLibraryImportPaths": [
      "/home/andrew/.local/share/swiftly/toolchains/6.0.3/usr/lib/swift/linux",
      "/home/andrew/.local/share/swiftly/toolchains/6.0.3/usr/lib/swift/linux/x86_64"
    ],
    "runtimeResourcePath": "/home/andrew/.local/share/swiftly/toolchains/6.0.3/usr/lib/swift"
  }

libTesting.so lives in `/path/to/toolchain/usr/lib/swift/linux".

So in order to find_library(SWIFT_TESTING NAMES Testing), one must first have a bunch of CMake goo to execute swiftc -print-target-info, parse the JSON for the items in the list runtimeLibraryImportPaths, and add them to a variable that can be used as a location for toolchain libs. At that point the project can "just"

include(swift-paths.cmake)
find_library(SWIFT_TESTING NAMES Testing PATHS ${SWIFT_LIBRARY_PATHS})

On macOS, there is no Testing.framework in the swift.org toolchain (as shipped by swiftly). In addition, the location of libTesting.dylib is no longer a path actually present in runtimeLibraryImportPaths of print-target-info. It's hidden in ".paths.runtimeLibraryImportPaths[0]" / testing:

  "paths": {
    "runtimeLibraryPaths": [
      "/Users/andrew/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2025-03-28-a.xctoolchain/usr/lib/swift/macosx",
      "/usr/lib/swift"
    ],
    "runtimeLibraryImportPaths": [
      "/Users/andrew/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2025-03-28-a.xctoolchain/usr/lib/swift/macosx"
    ],
    "runtimeResourcePath": "/Users/andrew/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2025-03-28-a.xctoolchain/usr/lib/swift"
  }

(venv) andrew@Andrews-MacBook-Pro-2:~/Source/ladybird-browser$ find ~/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2025-03-28-a.xctoolchain/ -name "Testing.framework" 
(venv) andrew@Andrews-MacBook-Pro-2:~/Source/ladybird-browser$ ls $(swiftc -print-target-info | jq -r ".paths.runtimeLibraryImportPaths[0]")/testing
Testing.swiftcrossimport    _TestDiscovery.swiftmodule    libTesting.dylib
Testing.swiftmodule        _Testing_Foundation.swiftmodule    lib_Testing_Foundation.dylib

If I just do a normal find_library on macOS when using a CMAKE_Swift_COMPILER that points to a swift.org toolchain from swiftly, it just finds Testing.framework in the MacOSX.platform path in Xcode.app. Which is definitely no the one I want to be using, especially if there's a bugfix on main of swift-testing that I need for local development.

This doesn't go into why I need to find libTestingMacros.so,dylib, but this comment is already pretty long...

ADKaster avatar Mar 31 '25 07:03 ADKaster

So I guess this is really more of a "everything is different depending on platform" issue looking at things more closely...

Xcode.app (16.2 16C5032a):

  • libTesting.dylib lives in /Applications/Xcode.app/Contents/Developer/Platform/MacOSX.platform/Developer/Library/Frameworks/Testing.framework
  • libTestingMacros.dylib lives in /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/host/plugins/testing

swiftly main-snapshot-2025-03-28 on macOS:

  • libTesting.dylib lives in ~/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2025-03-28-a.xctoolchain/usr/lib/swift/macosx/testing
  • libTestingMacros.dylib lives in ~/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2025-03-28-a.xctoolchain/usr/lib/swift/host/plugins/testing

swiftly main-snapshot-2025-03-28 on Linux:

  • libTesting.so lives in ~/.local/share/swiftly/toolchains/main-snapshot-2025-03-25/usr/lib/swift/linux/
  • libTestingMacros.so lives in ~/.local/share/swiftly/toolchains/main-snapshot-2025-03-25/usr/lib/swift/host/plugins

Having to account for all these paths in my CMake find module is tedious. It would be nicer if the toolchain provided me a find module in /usr/share/cmake or /usr/share/swift/cmake of the toolchain, but that's a separate discussion.

ADKaster avatar Mar 31 '25 08:03 ADKaster

Just so you're aware and as a data point, Testing.framework is Apple's fork of Swift Testing and its install path is not controlled by the Swift open source project.

Xcode itself has command-line tools like xcrun to help you find paths to its various bits and pieces.

grynspan avatar Mar 31 '25 13:03 grynspan

It will probably take a while to complete the relocation of this macro plugin across all platforms, but I've posted a Swift Package Manager PR (https://github.com/swiftlang/swift-package-manager/pull/8670) which should allow that transition to begin.

stmontgomery avatar May 14 '25 18:05 stmontgomery

Tracked by rdar://147985636

stmontgomery avatar Jul 01 '25 21:07 stmontgomery