Incompatible with Xcode 11b4 and xctestplans

[daveanderson]

Version 6.1.0 appears to be incompatible with Xcode 11b4 after enabling xctestplans on my project per the following error I am receiving.

xctest (40204) encountered an error (Failed to load the test bundle. (Underlying error: The bundle “PrivateFrameworkTests” couldn’t be loaded because it is damaged or missing necessary resources. The bundle is damaged or missing necessary resources. dlopen_preflight(/Users/user/Library/Developer/Xcode/DerivedData/PrivateApp/Build/Products/Test-iphonesimulator/PrivateFrameworkTests.xctest/PrivateFrameworkTests): Library not loaded: /usr/lib/swift/libswiftXCTest.dylib
  Referenced from: /Users/user/Library/Developer/Xcode/DerivedData/PrivateApp/Build/Products/Test-iphonesimulator/PrivateFrameworkTests.xctest/Frameworks/FBSnapshotTestCase.framework/FBSnapshotTestCase
  Reason: no suitable image found.

A sample project without ios-snapshot-test-case works with xctestplans but with test plans I'm unable to run my tests.

Disabling iOSSnapshotTestCase as a pod in my project allows my tests to proceed

[juhanh]

I am encountering the same error even without using xctestplans. Xcode 11b4 here, too.

[bielikb]

Im encountering the same issue with Xcode 11 beta 5, too. Will this be fixed by the time of the release of Xcode 11?

[lizjakubowski]

We found that setting this build setting in FBSnapshotTestCase fixes the test bundle launch errors:

LIBRARY_SEARCH_PATHS = $(TOOLCHAIN_DIR)/usr/lib/swift-$(SWIFT_VERSION)/$(PLATFORM_NAME) $(inherited)

Where TOOLCHAIN_DIR is the path to XcodeDefault.xctoolchain and PLATFORM_NAME is, e.g. iphoneos

[robmathers]

Also running into this issue (although we're on an older release), on the Xcode 11 GM. Unfortunately the LIBRARY_SEARCH_PATHS fix didn't get the tests running for us.

[juhanh]

Turns out that the culprit for our case was using iOS 12.0 sim which is broken. LIBRARY_SEACH_PATHS did fix the specific issue with iOSSnapshotTestCase but surfaced another similar one. Switching to iOS 12.2 sim works without any manual intervention.

[robmathers]

Good tip from @juhanh. Switching to the iOS 13 simulator (from 10.3) also did the trick. Unfortunately a lot of our tests break due to small rendering differences in the OS, so I'm still hoping there's a fix that allows us to use older OS simulators.

[reidmain]

I just downloaded the Xcode 11 GM seed 2 (11A420a) and converted the FBSnapshotTestCaseDemo project to use test plans. I didn't encounter any issues on an iPhone 8 simulator with both iOS 12.2 and 13.

Could anyone who has been having issues double check with the GM seed 2 to see if things have been fixed?

[ANGOmarcello]

Hey there, I can confirm that Xcode 11 GM 2 still has issues with loading or saving the reference images. Will get back later to you wether LIBRARY_SEACH_PATHS did fix the issue for us.

[amrox]

I'm seeing the same issue attempting to run tests against 11.4 simulators. I tried

LIBRARY_SEARCH_PATHS = $(TOOLCHAIN_DIR)/usr/lib/swift-$(SWIFT_VERSION)/$(PLATFORM_NAME) $(inherited)

I did want to add one thing. I was still using Swift 4.2, so the path resolved to...

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift-4.2/iphoneos

... which does not exist in Xcode 11 (11A419c). However, I changed everything to use Swift 5.0 and it still didn't work 😔

[reidmain]

Drat I am sorry to hear that. @ANGOmarcello, @amrox I had to ask this but if you clone this repository, open the FBSnapshotTestCaseDemo, run pod install, convert to test plans and then run them do you see the error?

Again I hate to ask this but I have yet to find a way to reproduce the issue locally so it is impossible for me to find a solution.

[ANGOmarcello]

I could trace the issue down to a recent change which coincidentally happend near the same time as us switching to Xcode 11. For us the issue was caused by using multiple schemes with shared snapshots.

The solution was to actually use the best practises for storing the images outlined by this repo.

[amrox]

@reidmain appreciate the input. Unfortunately, I'm not sure Test Plans will work for us, as we need to keep production builds on Xcode 10 for the time being. We provide a binary framework and usually have to support 1 Xcode version back for awhile to maintain bitcode compatibility while our clients upgrade.

I should have also mentioned that I'm seeing this issue on a different fork of ios-snapshot-test-case. This one appears to be a lot more active, so I'm going to try to switch over and see if that helps. It is still happens I'll try to put up a sample project.

[reidmain]

Much appreciated. We're committed to doing whatever we can to have iOSSnapshotTestCase work on the latest Xcode releases so please don't hesitate to post any of your findings.

[amrox]

Maybe this will help? https://github.com/amrox/Bug-FBSnapshotTestCase-Xcode11

Please let me know if I can clarify anything. I'm going to keep hacking on it as well.

[antranapp]

Any update on this issue? We are seeing this issue too with Xcode Version 11.1 (11A1027). We don't use Cocoapod but Carthage.

[reidmain]

Maybe this will help? https://github.com/amrox/Bug-FBSnapshotTestCase-Xcode11

Please let me know if I can clarify anything. I'm going to keep hacking on it as well.

This is really interesting in that it only seems to affect iOS 11.4 simulators. iOS 12 and 13 work fine so perhaps this is an Xcode bug?

Is anyone else on this thread having the issues with iOS 12 or higher?

[artyom-razinov]

@lizjakubowski Thank you very much! It did solve an issue in my own project that used XCTest as a dependency.

I switched everything to swift 5 and added this in my podspec:

    'LIBRARY_SEARCH_PATHS' => '$(TOOLCHAIN_DIR)/usr/lib/swift-$(SWIFT_VERSION)/$(PLATFORM_NAME) $(inherited)'

Full source:

Pod::Spec.new do |s|
  s.name                   = 'MixboxTestsFoundation'
  s.module_name            = s.name
  s.version                = '0.2.3'
  s.summary                = s.name
  s.homepage               = 'https://github.com/avito-tech/Mixbox'
  s.license                = 'MIT'
  s.author                 = { 'Hive of coders from Avito' => 'avito.ru' }
  s.source                 = { :git => 'https://github.com/avito-tech/Mixbox.git', :tag => "Mixbox-#{s.version}" }
  s.platform               = :ios, '9.0'
  s.ios.deployment_target  = "9.0"
  s.swift_version = '5.0'
  s.requires_arc           = true
  s.source_files           = 'Frameworks/TestsFoundation/**/*.{swift,h,m,sh}'
  s.framework              = 'CoreLocation'
  
  s.dependency 'MixboxArtifacts'
  s.dependency 'MixboxFoundation'
  s.dependency 'MixboxReporting'
  s.dependency 'MixboxUiKit'
  
  s.dependency 'SQLite.swift'
  
  s.frameworks = 'XCTest', 'XCTAutomationSupport'
  
  s.xcconfig = {
    'LIBRARY_SEARCH_PATHS' => '$(TOOLCHAIN_DIR)/usr/lib/swift-$(SWIFT_VERSION)/$(PLATFORM_NAME) $(inherited)',
    'LD_RUNPATH_SEARCH_PATHS' => '$(inherited) "$(PLATFORM_DIR)/Developer/Library/Frameworks" "$(PLATFORM_DIR)/Developer/Library/PrivateFrameworks"',
    'FRAMEWORK_SEARCH_PATHS' => '$(inherited) "$(PLATFORM_DIR)/Developer/Library/Frameworks" "$(PLATFORM_DIR)/Developer/Library/PrivateFrameworks"'
  }
end

I was experiencing same issue after starting to use Xcode 11. I had this issue with iOS 10/11/12, but not with iOS 13.

[artyom-razinov]

I checked that adding LIBRARY_SEARCH_PATHS alone without switching to Swift 5 did not help!

[jn-se]

Could this issue be related to macOS version? Does not work for us with simulator on macOS Catalina. Same project seems to work properly with simulator on macOS Mojave. (Xcode 11 in both cases, but maybe not exactly the same version)

[jn-se]

Could this issue be related to macOS version? Does not work for us with simulator on macOS Catalina. Same project seems to work properly with simulator on macOS Mojave. (Xcode 11 in both cases, but maybe not exactly the same version)

Related to the framework version. Seems to be fixed with version 6.2.0.

[pogramos]

I'm experiencing the same problem using it to test on iOS 11.4 simulators, but it works fine on iOS 12 or later Even after updating to 6.2.0

Project.app (20066) encountered an error (Failed to load the test bundle. 
(Underlying error: The bundle “ProjectTest” couldn’t be loaded because it is damaged or missing necessary resources. 
The bundle is damaged or missing necessary resources. 
   dlopen_preflight(/Users/user/Library/Developer/Xcode/DerivedData/Project-cptrilskmcwcukcunmexqnpkrula/Build/Products/Debug-iphonesimulator/Project.app/PlugIns/ProjectTests.xctest/ProjectTests): 
   Library not loaded: /usr/lib/swift/libswiftXCTest.dylib

  Referenced from: /Users/user/Library/Developer/Xcode/DerivedData/Project-cptrilskmcwcukcunmexqnpkrula/Build/Products/Debug-iphonesimulator/Project.app/PlugIns/ProjectTests.xctest/Frameworks/FBSnapshotTestCase.framework/FBSnapshotTestCase
  Reason: no suitable image found.  Did find:
	/usr/lib/swift/libswiftXCTest.dylib: mach-o, but not built for iOS simulator))

already cleared my build folder, derived data and so on.

[devedup]

I keep my reference images external to the default place using an environment variable:

FB_REFERENCE_IMAGE_DIR = $(SOURCE_ROOT)/$(PROJECT_NAME)SnapshotUITests/ReferenceImages

On migrating to Test Plans, I was experiencing the issues above and I had to re-enter my environment variables in .xctestplan >configuration > Environment variables.

After which, it started working again. 👍

[1oo7]

@pogramos FBSnapshotTestCase 5.0.2 can be made to successfully build and test in XCode 11 with any simulator device regardless of the OS.

Here is a regex to fix the pbxproj:

cat FBSnapshotTestCase.xcodeproj/project.pbxproj \
| tr "\n" "😁" | \
php -R ' 
$search1 = "/\"-framework\",😁\\t\\t\\t\\t\\tXCTest,😁/"; 
$replace1 = "\"-weak_framework\",😁\t\t\t\t\tXCTest,😁\t\t\t\t\t\"-weak-lXCTestSwiftSupport\",😁"; 
$replacement1 = preg_replace($search1, $replace1, $argn); 
$search2 = "/OTHER_LDFLAGS/"; 
$replace2 = "LIBRARY_SEARCH_PATHS = \"$(PLATFORM_DIR)/Developer/usr/lib\";😁\t\t\t\tOTHER_LDFLAGS"; 
$replacement2 = preg_replace($search2, $replace2, $replacement1);
print $replacement2;' \
| > temp; \
cat temp | tr "😁" "\\n" | \
> FBSnapshotTestCase.xcodeproj/project.pbxproj; \
rm temp

See also https://github.com/CocoaPods/CocoaPods/issues/9165#issuecomment-550273696

Basically Apple changed the dynamic libs for XCTest with XCode 11 / Swift 5.1. Older iOS device simulators may lack the right libraries inside their little brains. So you have to force them to know the New Truth, via yet more build setting cruft. It's a joy.

Oh and why did I resort to changing newlines to smiley faces? Because unix is from the days of the telegram where newlines literally had hardware implications. There is this hang-up it has about newlines. OS X sed doesn't handle multiline stuff without doing crazy things that no one can understand. It's much easier to just replace all newlines with some random character at the beginning, then do your regex as if the file was one big line, then restore all the newlines at the end.

Alternatively rather than modifying & rebuilding the project, you can simply copy over the new XCTest library your older simulator and create a symlink so it gets used (that's essentially what adding the above build-settings cruft does).

This did the trick for me, and is a lot more maintainable than modifying your carthage checkouts or whatever.

sudo zsh -c '
sourcedir="/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/usr/lib"; 
targetdir="/Library/Developer/CoreSimulator/Profiles/Runtimes/iOS 11.4.simruntime/Contents/Resources/RuntimeRoot/usr/lib"; 
cp $sourcedir/libXCTestSwiftSupport.dylib $targetdir; 
cp -R $sourcedir/XCTest.swiftmodule $targetdir; 
cp $sourcedir/libXCTestBundleInject.dylib $targetdir;
ln -s $targetdir/libXCTestSwiftSupport.dylib $targetdir/libswiftXCTest.dylib'