When using `upload-symbols` to upload dsyms to crashlytics, an error `No buffer space available` randomly shows that fails the upload

[acecilia]

Description

Hi:

When using upload-symbols (3.13 build 32) to upload dsyms to crashlytics, an error No buffer space available randomly shows that fails the upload. The rate of failure is something like once every 10 attempts. The detailed error shows when passing debug = true to upload-symbols.

The script is running on an M1 machine with MacOS version 12.6.1 (21G217).

Reproducing the issue

Cannot reproduce it consistently

Firebase SDK Version

N/A

Xcode Version

13.4.1

Installation Method

Carthage

Firebase Product(s)

Infrastructure

Targeted Platforms

macOS

Relevant Log Output

▸ Uploading cSYM for uuid xxx, architecture arm64
▸ error: Failed uploading cSYMs due to error: Error Domain=com.google.firebase.crashlytics.FIRCLSCompoundOperation.error Code=4294967293 "(null)" UserInfo={com.google.firebase.crashlytics.FIRCLSCompoundOperation.error.user-info-key.underlying-errors=(
▸ "Error Domain=com.crashlytics.mac.error-domain.submit-csym Code=0 \"Failed to submit cSYM for architecture arm64 with uuid xxx in dSYM: /var/folders/yf/2brc_wmj2f3dg85skf9ypc_m0000gq/T///Users/.../MyApp.app.dSYM.zip.unzipped/MyApp.app.dSYM\" UserInfo={NSLocalizedFailureReason=Failed to submit cSYM for architecture arm64 with uuid xxx in dSYM: /var/folders/yf/2brc_wmj2f3dg85skf9ypc_m0000gq/T///Users/.../MyApp.app.dSYM.zip.unzipped/MyApp.app.dSYM, NSUnderlyingError=0x600076e34030 {Error Domain=NSPOSIXErrorDomain Code=55 \"No buffer space available\" UserInfo={_NSURLErrorFailingURLSessionTaskErrorKey=LocalDataTask <1ADA85B6-6603-439C-8569-2B40F70FFF3B>.<6>, _kCFStreamErrorDomainKey=1, NSErrorPeerAddressKey=<CFData 0x600001f8dcc0 [0x1ea480000]>{length = 16, capacity = 16, bytes = 0x100201bbd83ad4ea0000000000000000}, _kCFStreamErrorCodeKey=55, _NSURLErrorRelatedURLSessionTaskErrorKey=(\n    \"LocalDataTask <1ADA85B6-6603-439C-8569-2B40F70FFF3B>.<6>\"\n)}}}"
▸ )}
▸ error: Could not complete submission of dSYM at /var/folders/yf/2brc_wmj2f3dg85skf9ypc_m0000gq/T///Users/MyApp.app.dSYM.zip.unzipped/MyApp.app.dSYM: Error Domain=com.google.firebase.crashlytics.FIRCLSCompoundOperation.error Code=4294967293 "(null)" UserInfo={com.google.firebase.crashlytics.FIRCLSCompoundOperation.error.user-info-key.underlying-errors=(
01:22:27   ▸ "Error Domain=com.crashlytics.mac.error-domain.submit-csym Code=0 \"Failed to submit cSYM for architecture arm64 with uuid xxx in dSYM: /var/folders/yf/2brc_wmj2f3dg85skf9ypc_m0000gq/T///Users/MyApp.app.dSYM.zip.unzipped/MyApp.app.dSYM\" UserInfo={NSLocalizedFailureReason=Failed to submit cSYM for architecture arm64 with uuid xxx in dSYM: /var/folders/yf/2brc_wmj2f3dg85skf9ypc_m0000gq/T///Users/.../MyApp.app.dSYM.zip.unzipped/MyApp.app.dSYM, NSUnderlyingError=0x600076e34030 {Error Domain=NSPOSIXErrorDomain Code=55 \"No buffer space available\" UserInfo={_NSURLErrorFailingURLSessionTaskErrorKey=LocalDataTask <1ADA85B6-6603-439C-8569-2B40F70FFF3B>.<6>, _kCFStreamErrorDomainKey=1, NSErrorPeerAddressKey=<CFData 0x600001f8dcc0 [0x1ea480000]>{length = 16, capacity = 16, bytes = 0x100201bbd83ad4ea0000000000000000}, _kCFStreamErrorCodeKey=55, _NSURLErrorRelatedURLSessionTaskErrorKey=(\n    \"LocalDataTask <1ADA85B6-6603-439C-8569-2B40F70FFF3B>.<6>\"\n)}}}"
▸ )}
Exit status of command '/Users/.../upload-symbols -d -gsp /Users/.../GoogleService-Info.plist -p ios /Users/MyApp.app.dSYM.zip' was 8 instead of 0.
upload-symbols 3.13 build 32

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[rizafran]

Thanks for reaching out, @acecilia. First off, could you provide the following details to check what's causing the issue:

• SDK version used
• Run script you've executed

Looking forward to your reply.

[acecilia]

First off, could you provide the following details to check what's causing the issue:

• SDK version used
• Run script you've executed

The SDK has nothing to do with this problem, I believe. When I need it, I download the latest version of upload-symbols binary from master and execute it.

The version of the binary where I am experiencing the problem is latest in master, which is 3.13 build 32.

The invocation I do is:

/Users/.../upload-symbols -d -gsp /Users/.../GoogleService-Info.plist -p ios /Users/.../MyApp.app.dSYM.zip

[acecilia]

I believe this may be the root cause for similar issues reported in fastlane:

• https://github.com/fastlane/fastlane/issues/15736
• https://github.com/fastlane/fastlane/issues/14140

[acecilia]

I can also reproduce this using upload-symbols 3.13 build 32 in Xcode 14.1 and MacOS 12.6.1-21G217

[acecilia]

@rizafran Sorry to ping again, but this is problematic as it results in random failures of the release job

[acecilia]

Is there any alternative to upload-symbols for uploading dsyms from a script?

[rizafran]

Sorry for the late revert, @acecilia. I'll be consulting this to the team and see what we can do. For the meantime, you may try uploading your dSYMs using one of the options here.

[samedson]

@acecilia I've been doing some research on this. One part of the error I'm researching is: Domain=NSPOSIXErrorDomain Code=55 \"No buffer space available\"

Based on that, it seems to be something on the OS that's failing. When researching that message I see suggestions of updating OS versions or checking whether the machine is out of memory:

• https://stackoverflow.com/questions/67318867/error-domain-nsposixerrordomain-code-28-no-space-left-on-device-userinfo-kcf
• https://developer.apple.com/forums/thread/714393

A couple areas to dig into

• What macOS version is the CI machine running?
• How much memory / disk space is left?
• Are there multiple instances of upload-symbols running?

[acecilia]

Thanks @samedson!

Here some more info:

What macOS version is the CI machine running?

MacOS 12.6.1. The issue I see happens in different agents, not limited to only one. All of the agents I have are M1

How much memory / disk space is left?

The issue happens when trying to upload dsyms after a build takes place, and these agents have average 100GB free disk space, so I would initially say lack of space is not a problem. If this was a problem, I would expect it failing during the build process, which is a more disk hungry operation.

Said that, I cannot know the free disk space at the time of the failure because I am not logging it. I can add logs, or if you prefer you can add them to upload-symbols under the debug option and I can paste here when I find failures, let me know.

Are there multiple instances of upload-symbols running?

No

[themiswang]

Hi @acecilia I will followup on this: Unfortunately current we don't open source upload-symbols cli so the change on that need a sdk update, to quick unblock on your side I would suggest to do a log on your side to double check. Right now I am using MacOS12.6.2 M1 for development and I have a version of xcode 13.4.1, so far I have not experiencing with this issue, can you try upgrading the OS and see if it still happen on your side?

[acecilia]

can you try upgrading the OS and see if it still happen on your side?

I will try, but it will take a while.

Have you seen the other linked reports in here?

[akhakimzyanov]

Hi! I have the some trouble. Is any workaround found?

[acecilia]

No workaround from my side so far

[BalestraPatrick]

This is a problem for us as well.

[themiswang]

Hi folks,

We still haven’t tracked this down to the root cause. Looking in the code, we compress the code mapping file and the create tasks for NSURLSession to upload the symbol. We set it up so that it does 1 session at a time, but maybe there’s a bug on our end that isn’t respecting that.

The other idea is I wonder if sockets from previous runs are being left open which cause "No buffer space available".

Next time you run into this, can you run lsof -i | egrep 'upload-sy' on the CI machine and see if there are open connections for upload-symbols? The socket for upload-symbol should look similar like this:

upload-sy 87805 yourName    9u  IPv4 0x5f099b59e899a12d      0t0  UDP 192.168.1.34:55180->bh-in-f95.1e100.net:https

[google-oss-bot]

Hey @acecilia. We need more information to resolve this issue but there hasn't been an update in 5 weekdays. I'm marking the issue as stale and if there are no new updates in the next 5 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

[acecilia]

Im on holidays at the moment. Will be able to test above when I’m back

[google-oss-bot]

Hey @acecilia. We need more information to resolve this issue but there hasn't been an update in 5 weekdays. I'm marking the issue as stale and if there are no new updates in the next 5 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

[acecilia]

Lets keep it open, still an issue. I added the lsof commands, need some time for the flakiness to appear to see if logs show something

[acecilia]

Since I updated to MacOS Ventura last week, I am not experiencing the problem anymore

[paulb777]

@acecilia Thanks for letting us know. I'll close for now and we can reopen if it reoccurs.