Feature Request: Swift Package Manager Precompiled Binaries

[xavierLowmiller]

Goals

We are in the process of adopting SPM as our primary package manager. Given that it now supports binary artefacts, is there any chance that Realm can be distributed as a precompiled binary? Compiling Realm is a good chunk of our clean build, and having precompiled binaries would probably speed things up for many developers using Realm and SPM.

Version of Realm and Tooling

Xcode version: 12+

iOS/OSX version: 11+

[tgoyne]

As of XCode 12.2 beta 3, SPM's binary artifact support doesn't actually work. You can build an application which depends on a prebuilt framework, but Archiving the application fails to set the search paths correctly and that step fails. I filed FB8706513 for this.

Once it actually works there still isn't a great way to let people pick at install time whether they want to use a binary or build from source, and there's some significant advantages to building from source.

[xavierLowmiller]

Very good points!

Maybe this can be revisited once the dust settles around SPM binary artefacts.

Thanks for your answer!

[gshahbazian]

@tgoyne Any chance this has been improved by SPM in Xcode 12.3? We would also like to move our packages to SPM and away from cocoapods but the clean build time here is a major drawback.

Can this issue be re-opened for tracking when Apple addresses the issue/radar?

[tgoyne]

If Apple does ever fix it I think it'd make sense to look into what we can offer. I think we actually could package just the obj-c xcframework in a spm-compatible format, which would eliminate the vast majority of the build time (at the cost of being a nontrivial download size).

[gshahbazian]

@tgoyne that would be great if do-able! Could the realm-core xcframework that the pod script is already using be used for this?

[tgoyne]

No, that's the thing that's broken. For various reasons we need the core library to be a static library when it's linked into the obj-c library, and consuming a static xcframework via spm makes it so that you can't archive the app. Consuming a dynamic xcframework also was broken in 12.0 but has now been fixed, so we could potentially ship a pre-build obj-c dynamic framework and only build the swift part from source.

[gshahbazian]

@tgoyne interesting. Thanks for the explanation. Would an xcframework for the objc library make clean builds from SPM faster than one from cocoapods?

[timstudt]

@tgoyne

.. and consuming a static xcframework via spm makes it so that you can't archive the app. Consuming a dynamic xcframework also was broken in 12.0 but has now been fixed, so we could potentially ship a pre-build obj-c dynamic framework and only build the swift part from source.

the archive issue is fixed in Xcode12.5 (see FB7608638, and https://developer.apple.com/documentation/xcode-release-notes/xcode-12_5-release-notes). Firebase SDK had the same issue and it's been closed now. it would be super great to use Realm as binary dependency via SPM, our build time increased significantly after adding Realm.

[denys-meloshyn]

Can it be reopened and fixed? 🙃

[leemaguire]

We are looking into this now.

[lkuczborski]

What is the current status of this? Would be great to have precompiled binaries in SPM.

[philprime]

I actually managed to get this working by downloading the release files, extracting the .xcframework files, individually compressing them separately, as you need the Realm.xcframework and the RealmSwift.xcframework as separate ZIP files, uploading them to my own cloud storage and declaring the as the following in the Package.swift:

    targets: [
        .binaryTarget(name: "Realm",
                      url: "https://my-cloud-storage/frameworks/Realm/10.11.0.zip",
                      checksum: "ededbea676d0acfd8136c40c22e733a1ed9a623a049c40782057789a474d2fb9"),
        .binaryTarget(name: "RealmSwift",
                      url: "https://my-cloud-storage/frameworks/RealmSwift/10.11.0.zip",
                      checksum: "b022fa4d881400249ee50348bf08d1e8625dc748ab5355f73b41e129566b4ad9")
    ]

I guess if the Realm team could additionally upload the Realm.xcframework and the RealmSwift.xcframework as own ZIP files in the releases, we could directly set the GitHub URL in the Package.swift manifest

[bioche]

While Realm team is sorting this out, I created a repo based on @philprime idea : https://github.com/bioche/RealmBinaries It stores xcframeworks in the release note for XCode 12.5.1 & XCode 13.0. You just have to reference RealmBinaries in the Package.swift like so (the tags will match Realm tags) :

.package(name: "RealmBinaries", url: "https://github.com/bioche/RealmBinaries", .upToNextMajor(from: "10.0.0"))

[chkpnt]

I can confirm @bioche's repo works like a charm. 👍 Looking forward for the integration in the official releases. :-)

[georgescumihai]

Using the workaround from @bioche made the preview work again, but we started to experience random failures on CI with no such module 'RealmSwift', not sure what is the issue, maybe SPM wasn't able to always download in time the prebuilt framework. In the end we reverted the change, but for a short amount of time preview worked :).

[bioche]

@georgescumihai I've seen it on my end also for one module ... It seems related to this thread : https://forums.swift.org/t/bug-with-binarytarget-in-swift-packages-with-xcode/45191.

On my particular case, I have an app AExample that depends on a local package A, that depends on a RealmHelper package that depends on RealmBinaries. It fails only on CI & I had 3 other example apps that depend on RealmBinaries with the same hierarchy that work just fine ^^

What I ended up doing is a quick hack : I kept the binaries on RealmBinaries github but deleted the dependency between RealmHelper & RealmBinaries to have Realm & RealmSwift binary targets directly in RealmHelper.

I think this is a weird XCode bug but I didn't test if XCode 13.1 actually fixed it ;)

[chkpnt]

Uh, just noticed in the release notes of v10.18.0: - Add prebuilt binary for Xcode 13.1 to the release package.

In the latest release notes (v10.20.2), the prebuilt binary is mentioned, too: Rebuild 10.20.1 with Xcode 13.2.1 rather than 13.2.0. This version has no changes if you are not using a prebuilt binary for Realm.

[chkpnt]

Mh.... Xcode is still compiling Realm when using SPM.

[tgoyne]

We have prebuilt xcframeworks available on each of the releases, which the spm package does not use.

[philprime]

@tgoyne even tough the xcframework you provide work fine, the asset archive structure is not suitable for SPM.

You are providing a single realm-swift-X.Y.Z.zip archive which contains the compiled .xcframework for each Xcode version (and the respective Swift version) combined..

The file tree looks like the following:

realm-swift-10.20.2.zip 
├── 12.4
│   ├── Realm.xcframework
│   └── RealmSwift.xcframework
├── 12.5.1
│   ├── Realm.xcframework
│   └── RealmSwift.xcframework
├── 13.0
│   ├── Realm.xcframework
│   └── RealmSwift.xcframework
├── 13.1
│   ├── Realm.xcframework
│   └── RealmSwift.xcframework
└── 13.2.1
    ├── Realm.xcframework
    └── RealmSwift.xcframework

But to be able to use precompiled frameworks with SPM, the respective .xcframework needs to be the top-level object of the archive. Furthermore the top-level object must be the exact name of the framework, as Xcode tries to copy the Realm.xcframework file from the archive which contains the Realm SPM package.

Therefore you need to publish them in individual release assets.

Example Archive Format:

realm-<variant>__<framework>__<version>__<Xcode>.xcframework.zip

Options:

• variant: objc or swift
• framework: Realm or RealmSwift
• version: usual semantic versioning
• Xcode: the Xcode version 12.4, 13.1 etc.

Something like the following tree for each release:

realm-swift__Realm__10.20.2__12.4.zip
└── Realm.xcframework
realm-swift__RealmSwift__10.20.2__12.4.zip
└── RealmSwift.xcframework
realm-swift__Realm__10.20.2__13.1.zip
└── Realm.xcframework
realm-swift__RealmSwift__10.20.2__13.1.zip
└── RealmSwift.xcframework
...

@bioche did exactly that (but only for the swift variant, without the version number and only the latest two Xcode versions)--> https://github.com/bioche/RealmBinaries/releases/tag/10.20.1

Of course this leads to more assets in the GitHub release artifacts, but this way it is possible to directly access a specific framework archive from SPM

[szotp]

I cannot explain how painful it is to compile Realm in low resource situations like cloud CI. If it's not a binary it takes like 30 minutes to compile. Not to mention weird quirks that carthage & rome have.

Please give us something that SPM can consume directly.

[szotp]

I have found a hack to use CocoaPods with Carthage.xframework provided by realm:

1. Add Realm.podspec to your repository

Pod::Spec.new do |spec|
    spec.name         = 'Realm'
    spec.version      = '10.25.1'
    spec.license      = '-'
    spec.homepage     = '-'
    spec.authors      = '-'
    spec.summary      = '-'
    spec.source       = { :http => 'https://github.com/realm/realm-swift/releases/download/v10.25.1/Carthage.xcframework.zip' }
    spec.vendored_frameworks = ['Realm.xcframework', 'RealmSwift.xcframework']
end

2. Add a pod to Podfile

pod 'Realm', :podspec => 'Realm.podspec'

[chkpnt]

I still wonder why this issue does not cause more outcry.

[bobergj]

I still wonder why this issue does not cause more outcry.

Probably because carthage exists. Our project continues to use carthage mainly due to this issue. Do we want to move off carthage? Yes!

[chkpnt]

@leemaguire commented on 19 May 2021: We are looking into this now.

Any update for us?

[romeouald]

Come on. Why is this still an open issue?

[RomanEsin]

Made a repo which helps to solve this problem :) https://github.com/RomanEsin/RealmBinary

[tgoyne]

Starting with the next release we'll be automatically publishing SPM-compatible zips to our github releases. We still need to add an official repo which consumes those binaries via a wrapper package.

[krris]

Amazing! Thanks for the info! 🙌

[salqueng]

TLDR; I tried SPM compatible xcframework zips, and it worked, but rejected by App Store.

After the release v10.42.0, which offers SPM compatible xcframework zips, I created a local Package.swift file and applied to my application.

Package.swift

// swift-tools-version:5.8

import PackageDescription

func buildTargets() -> [Target] {
  let baseURL = "https://github.com/realm/realm-swift/releases/download/v10.42.0"
  let realmChecksum = "7f9a35c79761daa3384c3d8bdc830092c086efc61922befda8fe24c6e78c3d1c"

  #if swift(>=5.9)
    let xcodeVersion = "15.0"
    let realmSwiftChecksum = "c115d1ee1a375d4904f48d4e0884bccc3148c53e159a34c9a4a44499d1d44475"
  #else
    let xcodeVersion = "14.3.1"
    let realmSwiftChecksum = "f31d284e334b98005d9c30a14eb967c042f6e7de963949284c1a38f3f4c83fe1"
  #endif

  return [
    .binaryTarget(
      name: "Realm",
      url: "\(baseURL)/Realm.xcframework.zip",
      checksum: realmChecksum
    ),
    .binaryTarget(
      name: "RealmSwift",
      url: "\(baseURL)/RealmSwift@\(xcodeVersion).xcframework.zip",
      checksum: realmSwiftChecksum
    ),
  ]
}

let package = Package(
  name: "RealmSPM",
  platforms: [
    .macOS(.v10_13),
    .iOS(.v11),
    .tvOS(.v11),
    .watchOS(.v4),
  ],
  products: [
    .library(
      name: "Realm",
      targets: ["Realm"]
    ),
    .library(
      name: "RealmSwift",
      targets: ["Realm", "RealmSwift"]
    ),
  ],
  targets: buildTargets()
)

My app was built and worked nicely on simulators and real devices, but it was rejected by App Store Connect due to the following reasons. I'm using tuist and Xcode 14.3.1.

Rejected by App Store Connect, because the app uses non-public API & invalid sdk value

ITMS-90338: Non-public API usage - The app references non-public symbols in Frameworks/Realm.framework/Realm: __availability_version_check. If method names in your source code match the private Apple APIs listed above, altering your method names will help prevent this app from being flagged in future submissions. In addition, note that one or more of the above APIs may be located in a static library that was included with your app. If so, they must be removed. For further information, visit the Technical Support Information at http://developer.apple.com/support/technical/

ITMS-90512: Invalid sdk value - The value provided for the sdk portion of LC_BUILD_VERSION in (Redacted).app/Frameworks/Realm.framework/Realm is 17.0 which is greater than the maximum allowed value of 16.6.

It looks like that "Realm.xcframework" is built with Xcode 15.0 beta, so App Store denied the binary. I don't know why __availability_version_check is used.