Runtime Error: symbol `_$s11AllKeyPathss0B12PathIterablePTl` Not Found on macOS

[xanderdunn]

macOS 11.0.1. Xcode 12.2 beta 3. Swift For TensorFlow 0.12. My project successfully builds and links, but I immediately get this runtime error when I'm running my unit tests:

...
[1146/1146] Linking MyModelAppPackageTests

* Build Completed!
error: terminated(1): /Library/Developer/Toolchains/swift-tensorflow-RELEASE-0.12.xctoolchain/usr/libexec/swift/pm/swiftpm-xctest-helper /Users/xander/dev/my_model/.build/x86_64-apple-macosx/debug/MyModelAppPackageTests.xctest /var/folders/zk/5rt0n9yx4p9frf4ybzt_mcz40000gn/T/TemporaryFile.CMRnsY output:
    error: unableToLoadBundle("/Users/xander/dev/my_model/.build/x86_64-apple-macosx/debug/MyModelAppPackageTests.xctest")
    2020-12-17 12:02:10.911 swiftpm-xctest-helper[30233:7363767] Error loading /Users/xander/dev/my_model/.build/x86_64-apple-macosx/debug/MyModelAppPackageTests.xctest/Contents/MacOS/MyModelAppPackageTests:  dlopen(/Users/xander/dev/myp_model/.build/x86_64-apple-macosx/debug/MyModelAppPackageTests.xctest/Contents/MacOS/MyModelAppPackageTests, 265): Symbol not found: _$s11AllKeyPathss0B12PathIterablePTl
      Referenced from: /Library/Developer/Toolchains/swift-tensorflow-RELEASE-0.12.xctoolchain/usr/lib/swift/macosx/libswiftTensorFlow.dylib
      Expected in: /usr/lib/swift/libswiftCore.dylib
     in /Library/Developer/Toolchains/swift-tensorflow-RELEASE-0.12.xctoolchain/usr/lib/swift/macosx/libswiftTensorFlow.dylib

It looks like it can't find the symbol _$s11AllKeyPathss0B12PathIterablePTl

Is it perhaps related to this code in my project:

public extension KeyPathIterable {
    /*
    This is based on clipByGlobalNorm here:
    https://github.com/tensorflow/swift-models/blob/542a03c09aa7bd6e326a18b5d086f2bec8b4b24c/Models/Text/BERT/Utilities.swift#L23
    Get all tensors in a gradient and get their min values, max values, and if any values are nan
    See a similar hasNaN callback here: https://github.com/tensorflow/swift-models/pull/526/files#diff-2b4e6467b43db097c85ce302b963fefe2ae32a9d42e7147143e0d088a5cc6c55R103
    */
    func gradientMinsAndMaxes<Scalar: TensorFlowFloatingPoint>() -> ([Scalar], [Scalar], Bool) {
        var mins: [Scalar] = []
        var maxes: [Scalar] = []
        var hasNan: Bool = false
        for kp in self.recursivelyAllWritableKeyPaths(to: Tensor<Scalar>.self) {
            let tensor: Tensor<Scalar> = self[keyPath: kp]
            mins.append(tensor.min().scalarized())
            maxes.append(tensor.max().scalarized())
            if hasNan == false {
                if tensor.isNaN.any() {
                    hasNan = true
                }
            }
        }
        return (mins, maxes, hasNan)
    }

    func numParamters<Scalar: TensorFlowFloatingPoint>() -> Scalar {
        var numParameters: Scalar = 0
        for kp in self.recursivelyAllWritableKeyPaths(to: Tensor<Scalar>.self) {
            let tensor: Tensor<Scalar> = self[keyPath: kp]
            numParameters += Scalar(tensor.shape.dimensions.cumulativeProduct())
        }
        return numParameters
    }
}

These KeyPathIterable extensions are very similar to swift-models code here and here.

• Note that this is specific to macOS. The same project using S4TF 0.12 on Linux does not produce this runtime error. The unit tests pass without issue.
• These unit tests do pass without issue on S4TF 0.11 on macOS, this is new with 0.12.

[xanderdunn]

Wow this appears to be specific to running the unit tests in parallel. When I execute swift test -Xswiftc -g -Xswiftc --target=x86_64-apple-macos10.15 -Xlinker -L/usr/local/lib, all tests complete without the runtime error.

When I execute swift test -Xswiftc -g -Xswiftc --target=x86_64-apple-macos10.15 -Xlinker -L/usr/local/lib --parallel --num-workers 6, I immediately get the above error.

It appears to be 100% reproducible. I ran with --parallel --num-workers 6 seven times and it always immediately produced the error. I ran withOUT --parallel --num-workers 6 twice and the tests completed both times. This takes over 15min so I only did it twice.

Note that this never occurs on Linux. My default of running the unit tests in parallel never produces this error.

[xanderdunn]

I updated my Package.swift to macOS 11:

    platforms: [
       .macOS(.v11)
    ],

and my build and test targets to macos11:

swift build -Xswiftc -g -Xswiftc --target=x86_64-apple-macos11 -Xlinker -L/usr/local/lib
swift test -Xswiftc -g -Xswiftc --target=x86_64-apple-macos11 -Xlinker -L/usr/local/lib

Same result. When I apply --parallel I get the runtime error and otherwise I don't get the runtime error. Adding a filter also hits the runtime error: --filter MyTestName. It doesn't matter what the test name is:

Testing with neither filter nor parallel works:

$ swift test -Xswiftc -g -Xswiftc --target=x86_64-apple-macos11 -Xlinker -L/usr/local/lib
Test Suite 'All tests' started at 2020-12-17 12:48:47.143
Test Suite 'MyModelAppPackageTests.xctest' started at 2020-12-17 12:48:47.144
Test Suite 'CompressionTests' started at 2020-12-17 12:48:47.144
Test Case '-[MyModelTests.CompressionTests testBitRepresentations]' started.
0100000000110110001100110011001100110011001100110011001100110011
1100000000110110010011001100110011001100110011001100110011001101
0100000000110110
8.1215e-320
1100000000110110
2.4311e-319
Test Case '-[MyModelTests.CompressionTests testBitRepresentations]' passed (0.055 seconds).
Test Case '-[MyModelTests.CompressionTests testLeadingBitsEntropy]' started.
Generating data...
Done
Uniform to pmf...
270 unique uniform leading bits values
Saving bar graph to ./leadingBitsUniform.png...
Saving histogram to ./leadingBitStringsUniform.png...
Done
uniform discrete entropy: 6.942957450193309
Normal to pmf...
323 unique normal leading bits values
Saving bar graph to ./leadingBitsNormal.png...
Saving histogram to ./leadingBitStringsNormal.png...
Done
normal discrete entropy: 7.429205662273593
Test Case '-[MyModelTests.CompressionTests testLeadingBitsEntropy]' passed (11.379 seconds).
Test Case '-[MyModelTests.CompressionTests testRandomCompressionRatio]' started.
Swift random uniform doubles compression ratio: 1.0573479071119865
Test Case '-[MyModelTests.CompressionTests testRandomCompressionRatio]' passed (0.591 seconds).
Test Suite 'CompressionTests' passed at 2020-12-17 12:48:59.171.
	 Executed 3 tests, with 0 failures (0 unexpected) in 12.026 (12.026) seconds
...

Filter can't find symbol:

$ swift test -Xswiftc -g -Xswiftc --target=x86_64-apple-macos11 -Xlinker -L/usr/local/lib --filter Compression
error: terminated(1): /Library/Developer/Toolchains/swift-tensorflow-RELEASE-0.12.xctoolchain/usr/libexec/swift/pm/swiftpm-xctest-helper /Users/xander/dev/My_model/.build/x86_64-apple-macosx/debug/MyModelAppPackageTests.xctest /var/folders/zk/5rt0n9yx4p9frf4ybzt_mcz40000gn/T/TemporaryFile.WhHN1A output:
    error: unableToLoadBundle("/Users/xander/dev/My_model/.build/x86_64-apple-macosx/debug/MyModelAppPackageTests.xctest")
    2020-12-17 12:48:07.549 swiftpm-xctest-helper[81536:7499645] Error loading /Users/xander/dev/My_model/.build/x86_64-apple-macosx/debug/MyModelAppPackageTests.xctest/Contents/MacOS/MyModelAppPackageTests:  dlopen(/Users/xander/dev/My_model/.build/x86_64-apple-macosx/debug/MyModelAppPackageTests.xctest/Contents/MacOS/MyModelAppPackageTests, 265): Symbol not found: _$s11AllKeyPathss0B12PathIterablePTl
      Referenced from: /Library/Developer/Toolchains/swift-tensorflow-RELEASE-0.12.xctoolchain/usr/lib/swift/macosx/libswiftTensorFlow.dylib
      Expected in: /usr/lib/swift/libswiftCore.dylib
     in /Library/Developer/Toolchains/swift-tensorflow-RELEASE-0.12.xctoolchain/usr/lib/swift/macosx/libswiftTensorFlow.dylib

Parallel can't find symbol:

$ swift test -Xswiftc -g -Xswiftc --target=x86_64-apple-macos11 -Xlinker -L/usr/local/lib --parallel
error: terminated(1): /Library/Developer/Toolchains/swift-tensorflow-RELEASE-0.12.xctoolchain/usr/libexec/swift/pm/swiftpm-xctest-helper /Users/xander/dev/My_model/.build/x86_64-apple-macosx/debug/MyModelAppPackageTests.xctest /var/folders/zk/5rt0n9yx4p9frf4ybzt_mcz40000gn/T/TemporaryFile.HfIqTi output:
    error: unableToLoadBundle("/Users/xander/dev/My_model/.build/x86_64-apple-macosx/debug/MyModelAppPackageTests.xctest")
    2020-12-17 12:49:41.126 swiftpm-xctest-helper[87127:7512023] Error loading /Users/xander/dev/My_model/.build/x86_64-apple-macosx/debug/MyModelAppPackageTests.xctest/Contents/MacOS/MyModelAppPackageTests:  dlopen(/Users/xander/dev/My_model/.build/x86_64-apple-macosx/debug/MyModelAppPackageTests.xctest/Contents/MacOS/MyModelAppPackageTests, 265): Symbol not found: _$s11AllKeyPathss0B12PathIterablePTl
      Referenced from: /Library/Developer/Toolchains/swift-tensorflow-RELEASE-0.12.xctoolchain/usr/lib/swift/macosx/libswiftTensorFlow.dylib
      Expected in: /usr/lib/swift/libswiftCore.dylib
     in /Library/Developer/Toolchains/swift-tensorflow-RELEASE-0.12.xctoolchain/usr/lib/swift/macosx/libswiftTensorFlow.dylib

So strange. 🤔

[xanderdunn]

I am sometimes seeing this without the --parallel or --filter additions to swift test, so it may be some kind of race condition. The error references Expected in: /usr/lib/swift/libswiftCore.dylib. Is this correct? I would expect the path to be the libswiftCore.dylib inside the S4TF toolchain. I have this set in my environment: export DYLD_LIBRARY_PATH=/Library/Developer/Toolchains/swift-tensorflow-RELEASE-0.12.xctoolchain/usr/lib/swift/macosx:"${DYLD_LIBRARY_PATH}".

[dan-zheng]

Thanks for reporting Xander!

The Symbol not found: _$s11AllKeyPathss0B12PathIterablePTl linker issue is related to ad-hoc additions to the Swift standard library on apple/swift:tensorflow branch.

We're currently making a push to de-fork the Swift compiler and use standard toolchains for building Swift for TensorFlow. Here's some more context. When that effort is done, this linker issue should naturally go away!

[xanderdunn]

@dan-zheng De-forking will be huge! Thanks for the context. I still find it curious that this first started appearing in 0.12 but never happened with 0.11, but I'm satisfied to test this again after the de-forking is complete. For now I'm able to run my tests on Linux only. Feel free to close or keep open until de-forking happens.

[FruityLeo]

Hi, I have just tried to install and run my first scripts (Usage.md) and the compiler is also not able to find the symbol s11AllKeyPathss0B12PathIterablePTl.

Build was successful but running the executable script gives the error below. Running the script with swift -O inference.swift works flawlessly.

I am also on macOs 11.0.1, Xcode 12.3 and followed the instructions from the guidelines.

Is this related or did I made a mistake when installing?