firebase-ios-sdk icon indicating copy to clipboard operation
firebase-ios-sdk copied to clipboard

Published 20 hours ago • verified publisher icon firebase

Upload-symbols has started to fail after update to Xcode14

Open niels-frydenholm opened this issue 1 year ago • 10 comments

Description

We have updated to Xcode14, and with that we are now having problems uploading symbols to crashlytics. The "fun" part is that it only fails on our CI server, whereas it's still possible to run same command locally.

Usually we use the fastlane action to do the upload, but to see if we could get more info about the error, we have tried a manual shellscript step which still leads to the same error.

Failed to upload symbols Exit status of command '[fullPath]/Pods/FirebaseCrashlytics/upload-symbols -d -gsp [fullPath]/Source/assets/firebase/GoogleService-Info.plist -p ios [fullPath]/appName.app.dSYM.zip' was instead of 0.

One thing that stands out is that there is no exit status - it's just a blank. I have tried to force other errors - e.g. not specifying the -gsp or use invalid credentials, and those errors show up nicely with more valid error info, so both the plist and the dSYM.zip should be valid.

Any chance you can provide some insights/hints to what can be causing a blank. exit status from the upload-symbols?

Reproducing the issue

Haven't been able to identify how to reproduce it locally - it's only on CI it behaves like this.

Firebase SDK Version

9.6.0

Xcode Version

14.0

Installation Method

CocoaPods

Firebase Product(s)

Analytics, App Distribution, Crashlytics, Messaging, Performance, Remote Config

Targeted Platforms

iOS

Relevant Log Output

No response

If using Swift Package Manager, the project's Package.resolved

Expand Package.resolved snippet 

Replace this line with the contents of your Package.resolved.

If using CocoaPods, the project's Podfile.lock

Expand Podfile.lock snippet 

PODS:
  - CTPanoramaView (1.5)
  - DynamicBlurView (4.1.0)
  - EcgObjCLibrary (1.2.0):
    - FLKAutoLayout (= 1.0.1)
    - Objection (= 1.6.1)
    - OCMock (~> 3)
    - PDKeychainBindingsController (= 0.0.1)
  - EcgObjCTestLibrary (0.2.0):
    - Objection (= 1.6.1)
    - OCMock (~> 3)
  - Firebase/Analytics (9.6.0):
    - Firebase/Core
  - Firebase/Core (9.6.0):
    - Firebase/CoreOnly
    - FirebaseAnalytics (~> 9.6.0)
  - Firebase/CoreOnly (9.6.0):
    - FirebaseCore (= 9.6.0)
  - Firebase/Crashlytics (9.6.0):
    - Firebase/CoreOnly
    - FirebaseCrashlytics (~> 9.6.0)
  - Firebase/Messaging (9.6.0):
    - Firebase/CoreOnly
    - FirebaseMessaging (~> 9.6.0)
  - Firebase/Performance (9.6.0):
    - Firebase/CoreOnly
    - FirebasePerformance (~> 9.6.0)
  - Firebase/RemoteConfig (9.6.0):
    - Firebase/CoreOnly
    - FirebaseRemoteConfig (~> 9.6.0)
  - FirebaseABTesting (9.6.0):
    - FirebaseCore (~> 9.0)
  - FirebaseAnalytics (9.6.0):
    - FirebaseAnalytics/AdIdSupport (= 9.6.0)
    - FirebaseCore (~> 9.0)
    - FirebaseInstallations (~> 9.0)
    - GoogleUtilities/AppDelegateSwizzler (~> 7.7)
    - GoogleUtilities/MethodSwizzler (~> 7.7)
    - GoogleUtilities/Network (~> 7.7)
    - "GoogleUtilities/NSData+zlib (~> 7.7)"
    - nanopb (< 2.30910.0, >= 2.30908.0)
  - FirebaseAnalytics/AdIdSupport (9.6.0):
    - FirebaseCore (~> 9.0)
    - FirebaseInstallations (~> 9.0)
    - GoogleAppMeasurement (= 9.6.0)
    - GoogleUtilities/AppDelegateSwizzler (~> 7.7)
    - GoogleUtilities/MethodSwizzler (~> 7.7)
    - GoogleUtilities/Network (~> 7.7)
    - "GoogleUtilities/NSData+zlib (~> 7.7)"
    - nanopb (< 2.30910.0, >= 2.30908.0)
  - FirebaseCore (9.6.0):
    - FirebaseCoreDiagnostics (~> 9.0)
    - FirebaseCoreInternal (~> 9.0)
    - GoogleUtilities/Environment (~> 7.7)
    - GoogleUtilities/Logger (~> 7.7)
  - FirebaseCoreDiagnostics (9.6.0):
    - GoogleDataTransport (< 10.0.0, >= 9.1.4)
    - GoogleUtilities/Environment (~> 7.7)
    - GoogleUtilities/Logger (~> 7.7)
    - nanopb (< 2.30910.0, >= 2.30908.0)
  - FirebaseCoreInternal (9.6.0):
    - "GoogleUtilities/NSData+zlib (~> 7.7)"
  - FirebaseCrashlytics (9.6.0):
    - FirebaseCore (~> 9.0)
    - FirebaseInstallations (~> 9.0)
    - GoogleDataTransport (< 10.0.0, >= 9.1.4)
    - GoogleUtilities/Environment (~> 7.7)
    - nanopb (< 2.30910.0, >= 2.30908.0)
    - PromisesObjC (~> 2.1)
  - FirebaseInstallations (9.6.0):
    - FirebaseCore (~> 9.0)
    - GoogleUtilities/Environment (~> 7.7)
    - GoogleUtilities/UserDefaults (~> 7.7)
    - PromisesObjC (~> 2.1)
  - FirebaseMessaging (9.6.0):
    - FirebaseCore (~> 9.0)
    - FirebaseInstallations (~> 9.0)
    - GoogleDataTransport (< 10.0.0, >= 9.1.4)
    - GoogleUtilities/AppDelegateSwizzler (~> 7.7)
    - GoogleUtilities/Environment (~> 7.7)
    - GoogleUtilities/Reachability (~> 7.7)
    - GoogleUtilities/UserDefaults (~> 7.7)
    - nanopb (< 2.30910.0, >= 2.30908.0)
  - FirebasePerformance (9.6.0):
    - FirebaseCore (~> 9.0)
    - FirebaseInstallations (~> 9.0)
    - FirebaseRemoteConfig (~> 9.0)
    - GoogleDataTransport (< 10.0.0, >= 9.1.4)
    - GoogleUtilities/Environment (~> 7.7)
    - GoogleUtilities/ISASwizzler (~> 7.7)
    - GoogleUtilities/MethodSwizzler (~> 7.7)
    - nanopb (< 2.30910.0, >= 2.30908.0)
  - FirebaseRemoteConfig (9.6.0):
    - FirebaseABTesting (~> 9.0)
    - FirebaseCore (~> 9.0)
    - FirebaseInstallations (~> 9.0)
    - GoogleUtilities/Environment (~> 7.7)
    - "GoogleUtilities/NSData+zlib (~> 7.7)"
  - FLKAutoLayout (1.0.1)
  - GCDWebServer (3.5.4):
    - GCDWebServer/Core (= 3.5.4)
  - GCDWebServer/Core (3.5.4)
  - Google-Mobile-Ads-SDK (9.5.0):
    - GoogleAppMeasurement (< 10.0, >= 7.0)
    - GoogleUserMessagingPlatform (>= 1.1)
  - GoogleAnalytics (3.20.0)
  - GoogleAppMeasurement (9.6.0):
    - GoogleAppMeasurement/AdIdSupport (= 9.6.0)
    - GoogleUtilities/AppDelegateSwizzler (~> 7.7)
    - GoogleUtilities/MethodSwizzler (~> 7.7)
    - GoogleUtilities/Network (~> 7.7)
    - "GoogleUtilities/NSData+zlib (~> 7.7)"
    - nanopb (< 2.30910.0, >= 2.30908.0)
  - GoogleAppMeasurement/AdIdSupport (9.6.0):
    - GoogleAppMeasurement/WithoutAdIdSupport (= 9.6.0)
    - GoogleUtilities/AppDelegateSwizzler (~> 7.7)
    - GoogleUtilities/MethodSwizzler (~> 7.7)
    - GoogleUtilities/Network (~> 7.7)
    - "GoogleUtilities/NSData+zlib (~> 7.7)"
    - nanopb (< 2.30910.0, >= 2.30908.0)
  - GoogleAppMeasurement/WithoutAdIdSupport (9.6.0):
    - GoogleUtilities/AppDelegateSwizzler (~> 7.7)
    - GoogleUtilities/MethodSwizzler (~> 7.7)
    - GoogleUtilities/Network (~> 7.7)
    - "GoogleUtilities/NSData+zlib (~> 7.7)"
    - nanopb (< 2.30910.0, >= 2.30908.0)
  - GoogleDataTransport (9.2.0):
    - GoogleUtilities/Environment (~> 7.7)
    - nanopb (< 2.30910.0, >= 2.30908.0)
    - PromisesObjC (< 3.0, >= 1.2)
  - GoogleTagManager (7.4.1):
    - FirebaseAnalytics (< 10.0, >= 8.0)
    - GoogleAnalytics (~> 3.20)
  - GoogleUserMessagingPlatform (2.0.1)
  - GoogleUtilities/AppDelegateSwizzler (7.8.0):
    - GoogleUtilities/Environment
    - GoogleUtilities/Logger
    - GoogleUtilities/Network
  - GoogleUtilities/Environment (7.8.0):
    - PromisesObjC (< 3.0, >= 1.2)
  - GoogleUtilities/ISASwizzler (7.8.0)
  - GoogleUtilities/Logger (7.8.0):
    - GoogleUtilities/Environment
  - GoogleUtilities/MethodSwizzler (7.8.0):
    - GoogleUtilities/Logger
  - GoogleUtilities/Network (7.8.0):
    - GoogleUtilities/Logger
    - "GoogleUtilities/NSData+zlib"
    - GoogleUtilities/Reachability
  - "GoogleUtilities/NSData+zlib (7.8.0)"
  - GoogleUtilities/Reachability (7.8.0):
    - GoogleUtilities/Logger
  - GoogleUtilities/UserDefaults (7.8.0):
    - GoogleUtilities/Logger
  - HMSegmentedControl (1.5.6)
  - IGListDiffKit (4.0.0)
  - IGListKit (4.0.0):
    - IGListDiffKit (= 4.0.0)
  - nanopb (2.30909.0):
    - nanopb/decode (= 2.30909.0)
    - nanopb/encode (= 2.30909.0)
  - nanopb/decode (2.30909.0)
  - nanopb/encode (2.30909.0)
  - Objection (1.6.1)
  - OCMock (3.4.3)
  - PDKeychainBindingsController (0.0.1)
  - PopupDialog (1.1.1):
    - DynamicBlurView (~> 4.0)
  - PrebidMobile (1.12):
    - PrebidMobile/core (= 1.12)
  - PrebidMobile/core (1.12)
  - PromisesObjC (2.1.1)
  - RangeSeekSlider (1.9.0)
  - SBTUITestTunnelClient (8.3.0):
    - SBTUITestTunnelCommon
  - SBTUITestTunnelCommon (8.3.0)
  - SBTUITestTunnelServer (8.3.0):
    - GCDWebServer
    - SBTUITestTunnelCommon
  - SKPhotoBrowser (7.1.0)
  - UIImageViewAlignedSwift (0.8.2)

DEPENDENCIES:
  - CTPanoramaView (= 1.5)
  - EcgObjCLibrary (from `PrivatePods/EcgObjCLibrary`)
  - EcgObjCTestLibrary (from `PrivatePods/EcgObjCTestLibrary`)
  - Firebase/Analytics (= 9.6.0)
  - Firebase/Core (= 9.6.0)
  - Firebase/Crashlytics (= 9.6.0)
  - Firebase/Messaging (= 9.6.0)
  - Firebase/Performance (= 9.6.0)
  - Firebase/RemoteConfig (= 9.6.0)
  - FLKAutoLayout (= 1.0.1)
  - GCDWebServer
  - Google-Mobile-Ads-SDK (= 9.5.0)
  - GoogleTagManager (= 7.4.1)
  - HMSegmentedControl (= 1.5.6)
  - IGListKit (~> 4.0)
  - Objection (= 1.6.1)
  - OCMock (~> 3.4.3)
  - PDKeychainBindingsController (~> 0.0)
  - PopupDialog (= 1.1.1)
  - PrebidMobile (= 1.12)
  - RangeSeekSlider (from `https://github.com/technology-ebay-de/RangeSeekSlider.git`, tag `1.9.0`)
  - SBTUITestTunnelClient (= 8.3.0)
  - SBTUITestTunnelServer (= 8.3.0)
  - SKPhotoBrowser (= 7.1.0)
  - UIImageViewAlignedSwift (= 0.8.2)

SPEC REPOS:
  trunk:
    - CTPanoramaView
    - DynamicBlurView
    - Firebase
    - FirebaseABTesting
    - FirebaseAnalytics
    - FirebaseCore
    - FirebaseCoreDiagnostics
    - FirebaseCoreInternal
    - FirebaseCrashlytics
    - FirebaseInstallations
    - FirebaseMessaging
    - FirebasePerformance
    - FirebaseRemoteConfig
    - FLKAutoLayout
    - GCDWebServer
    - Google-Mobile-Ads-SDK
    - GoogleAnalytics
    - GoogleAppMeasurement
    - GoogleDataTransport
    - GoogleTagManager
    - GoogleUserMessagingPlatform
    - GoogleUtilities
    - HMSegmentedControl
    - IGListDiffKit
    - IGListKit
    - nanopb
    - Objection
    - OCMock
    - PDKeychainBindingsController
    - PopupDialog
    - PrebidMobile
    - PromisesObjC
    - SBTUITestTunnelClient
    - SBTUITestTunnelCommon
    - SBTUITestTunnelServer
    - SKPhotoBrowser
    - UIImageViewAlignedSwift


SPEC CHECKSUMS:
  CTPanoramaView: f7af526aaffedb2d52e87112ee0d57bb115eb6a2
  DynamicBlurView: 58e18fae80bb614e34681a4486870e7d257b62e8
  Firebase: 5ae8b7cf8efce559a653aef0ad95bab3f427c351
  FirebaseABTesting: 61826730ce9eee8781ba99a2b3420e9bce148dc9
  FirebaseAnalytics: 89ad762c6c3852a685794174757e2c60a36b6a82
  FirebaseCore: 2082fffcd855f95f883c0a1641133eb9bbe76d40
  FirebaseCoreDiagnostics: 99a495094b10a57eeb3ae8efa1665700ad0bdaa6
  FirebaseCoreInternal: bca76517fe1ed381e989f5e7d8abb0da8d85bed3
  FirebaseCrashlytics: 3210572ddb77801e5a0bd9d7bc890769f2066a0c
  FirebaseInstallations: 0a115432c4e223c5ab20b0dbbe4cbefa793a0e8e
  FirebaseMessaging: a4d7910e4af663c9cbfc1071c5bef34651690949
  FirebasePerformance: f1e62598cd98c88ce6c10e0d26223d18b85c26d6
  FirebaseRemoteConfig: ee09d77a7d7c7e31da6a0d1cf956cd611c85609c
  FLKAutoLayout: 37e1e09de6411dbee5526860d9f55d9063323ea8
  GCDWebServer: 2c156a56c8226e2d5c0c3f208a3621ccffbe3ce4
  Google-Mobile-Ads-SDK: caf686ba55108412e02a3eedb136e12e488dbfaa
  GoogleAnalytics: 01e4c5a04544a3608f93e970ce0817836897c5a5
  GoogleAppMeasurement: 6de2b1a69e4326eb82ee05d138f6a5cb7311bcb1
  GoogleDataTransport: 1c8145da7117bd68bbbed00cf304edb6a24de00f
  GoogleTagManager: 4a2104b801c150ad1e1cf64f3a2d78f8110b1125
  GoogleUserMessagingPlatform: 5f8b30daf181805317b6b985bb51c1ff3beca054
  GoogleUtilities: 1d20a6ad97ef46f67bbdec158ce00563a671ebb7
  HMSegmentedControl: 34c1f54d822d8308e7b24f5d901ec674dfa31352
  IGListDiffKit: 665d6cf43ce726e676013db9c7d6c4294259b6b2
  IGListKit: fd5a5d21935298f5849fa49d426843cff97b77c7
  nanopb: b552cce312b6c8484180ef47159bc0f65a1f0431
  Objection: ed03f2219cb22d2b5b893114219fe6c7e3263bde
  OCMock: 43565190abc78977ad44a61c0d20d7f0784d35ab
  PDKeychainBindingsController: fa8cb3cf99f2ea9240d61066ed0549f34e2cec3c
  PopupDialog: 720c92befd8bc23c13442254945213db5612f149
  PrebidMobile: 1c85021dc895419c0d205cf9e985c586a0152053
  PromisesObjC: ab77feca74fa2823e7af4249b8326368e61014cb
  SBTUITestTunnelClient: 873869031cec61a5b5540d66f5171a7abdad835d
  SBTUITestTunnelCommon: 21371c4796ce6826b937d29c79cd8c758f31d775
  SBTUITestTunnelServer: cb25d44ba2432d46efed49a0c3879c3d59f0915c
  SKPhotoBrowser: cd95b95a6f352fc2a50992721299f75ead77f9ab
  UIImageViewAlignedSwift: 12b0b06da988015e6d5dec09ee29d5ae1a244848

PODFILE CHECKSUM: aedbbec9a02834b6187d460a6deb407518e0966f

COCOAPODS: 1.11.3

niels-frydenholm avatar Sep 22 '22 15:09 niels-frydenholm

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

google-oss-bot avatar Sep 22 '22 15:09 google-oss-bot

I should note I have also tried the following:

  • get the dsym.zip from the CI server and then use upload-symbols with that zip on my local machine. That went well, so it's not like the zip is corrupt.
  • Tried to omit e.g. the plist, which immediately gives an error with details info on what's wrong.

Also, when it fails on CI, it does so immediately (within a second), so it seems like it's some prerequisite being wrong, but I'm running out of ideas of what to try.

niels-frydenholm avatar Sep 22 '22 17:09 niels-frydenholm

Thanks for reporting, @niels-frydenholm. Is the issue only happening on Xcode 14? Additionally, could you check the following:

  • Make sure that the DWARF with dSYM file is set for both debug and release builds.
  • Make sure that the upload_symbols_to_crashlytics action is added in the Fastfile. This should allow you to symbolicate the fastlane generated .ipa crashes.
  • You could try removing the dSYM path from fastlane so it will detect the path automatically, and see if that resolves the issue.

If that doesn’t work, could you provide the following info for me to get a better view of your issue:

rizafran avatar Sep 26 '22 17:09 rizafran

Yes, this problem has just started after we changed to use Xcode14. The same Fastlane setup build on the same Travis nodes works fine when running with an Xcode13 image.

DWARF with dsym is set for both debug and release (it was set to only DWARF for debug on simulators (to save build time), but I have tried to change that setting back to default, but it provides the same result.

I'm not completely sure what you mean with the second bullet about the upload_symbols_to_crashlytics is added to the Fastfile?

I have also tried to omit the dSYM path and let Fastlane detect it - it found the same zip, and got the same result.

I'm fairly certain the issue is not related to Fastlane, as I have tried to run the same upload-symbols command directly wrapped in a sh("...") step in the fastfile, which gives the same negative result.

I have tried fastlane with the verbose flag, but it doesn't provide any more info around the place where it fails (see the attached photo, where Fastlane continues after the upload_symbols_to_crashlytics - probably because it doesn't understand the blank statusCode coming back from the upload_symbols program)

A current workaround is that I upload the dSYM.zip file to Artifactory (because the Travis build instance is cleaned after use), and from there I can get the file and manually upload it to Crashlytics.

But I'm still scratching my head as to how the upload_symbols fails so fast, when the provided input parameters seems valid.

image

niels-frydenholm avatar Sep 28 '22 11:09 niels-frydenholm

The other weird thing is upload script reports success 13:20:28 Successfully submitted symbols for architecture arm64 with UUID f0d0b5c234de3712be84eca1d2ea913a in dSYM: /Users/XXX/.jenkins/workspace/XXX-TEST-DEVELOP-TestFlight/build/XXX-TEST-Release.xcarchive/dSYMs/XXX.framework.dSYM and 13:20:28 [32mSuccessfully uploaded Crashlytics symbols[0m But I keep receiving email with Crashlytics detected a missing dSYM for version 1.0 (0) while my version is 1.0.220.

dmi3j avatar Sep 30 '22 17:09 dmi3j

Thanks for the additional info, @niels-frydenholm. To narrow down and isolate if the issue is SDK-related, I just want to confirm if you have tried to manually upload the dSYMs using any of the options here? If so, is the issue also reproducible?

@dmi3j, it seems that the issue is on the backend side since the dSYM was successfully uploaded based on your logs. With this, please file a ticket through our support channel so they could check and investigate your project.

rizafran avatar Oct 03 '22 18:10 rizafran

I have tried to take the Zip file from the build-server (the same zip that fails to upload during build) and manually upload it to Crashlytics from the Browser. But it's a good point to try and take the same zip file and use the upload-symbols locally on my machine. I'll give that a go tomorrow and report back how it goes...

niels-frydenholm avatar Oct 04 '22 15:10 niels-frydenholm

I have now tried to use the zip file from the build-server, and uploading it with the upload-symbols from my local machine. That works fine, which adds to my own theory that "something" might have changed on our build-nodes, with the upgrade to Xcode14.

But with a "blank" errorCode from the upload-symbols I can only guess - e.g. is it some wrong version of a dependency, some file access problems, or something completely different 🤔 🤷

niels-frydenholm avatar Oct 05 '22 08:10 niels-frydenholm

I'm not super familiar with Travis, but it does seem like something about the environment on those xcode 14 machines has changed. I'm wondering if it could be M1 related?

  • It's interesting that Travis doesn't show Xcode 14 in their list here: https://docs.travis-ci.com/user/reference/osx/. Am I looking in the wrong place? I'm hoping to find some sort of changelog for their Xcode 14
  • Is there a way to get more information from the logs that Travis is spitting out? It seems likely that upload-symbols is crashing

samedson avatar Oct 07 '22 13:10 samedson

I forgot to note that we are running a self-hosted enterprise setup of Travis, but I don't have any real insights into the provisioned machines we are running on (although I know they are not M1, but my own dev machine is). I'll see if I can find out how to ssh into a machine (before it's closing down after build), and try various commands, that might help verify what could have changed.

niels-frydenholm avatar Oct 07 '22 16:10 niels-frydenholm

Hey folks, I am going to close this issue for now. Can you reopen if you are still running into problems?

samedson avatar Apr 27 '23 17:04 samedson