Symbolication crash reports issue

[vadimue]

Hi,

I can't see fully symbolized crash report in Rollbar. I tried to upload dSYM files to Rollbar settings in different ways:

1. Manually AppName.app.dSYM
2. Manually AppName.app.dSYM.zip
3. Manually dSYMs.zip (zipped folder with app dSYM and all frameworks)
4. Automatically by python script. (like in Rollbar docs https://rollbar.com/blog/ios-error-monitoring) Btw this script loads exactly same zip like in the case 2.

I found dSYMs from XCode -> Window -> Archives -> Show In Finder -> Show Package Contents on xcarchive -> dSYMs.

• I built a release app (Ad-hoc) and tried to crash app on the device. No results (Unsymbolicated report)
• I build debug app and tried to crash app on the simulator. Partially Symbolicated (I see only system calls, not call in my app)

Also, if I upload crash report from device directly to XCode - report fully symbolicated.

Any advice will help a lot, thanks!

[akornich]

@vadimue , I tried to repro the issue you described and it looks like it could be a backend processing issue (not this SDK issue). We will have a backend engineer to debug this scenario and see what could be breaking the symbolication process there.

[akornich]

#52

[letaniaferreira]

another customer affected - https://app.intercom.io/a/apps/rlisg82q/inbox/inbox/813342/conversations/24048857931

[luandrecast-lp]

experiencing the same issue

[iveo-team]

Still an issue: https://rollbar.com/iveo-contact/PROD-iOS/items/3/occurrences/139338831681/ Disappointing.

[akornich]

@iveo-team , it looks like you specified client.app.code_version as Version when uploading your dSYM. Instead, try using the client.app.short_version (i.e 1.5).

[francois-iveo]

@akornich: (speaking for iveo-team) I reuploaded the same dSYM with both the build version number as well as the public version number. Let's hope this solves it (although I have my doubts).

Another perhaps related matter is that unrelated exceptions are grouping together. See: https://rollbar.com/iveo-contact/PROD-iOS/items/3/?item_page=0

[akornich]

@francois-iveo , thanks for trying it out. Keep me in the loop about the outcome. I forwarded the issues of wrong grouping to our analytics team to look into it...

[francois-iveo]

@akornich: Behavior is still the same. See https://rollbar.com/iveo-contact/PROD-iOS/items/3/occurrences/149521530180/

Let me reiterate what the behavior is:

1. Initially Rollbar reports 'This backtrace could not be symbolicated because a matching dSYM upload was not present'
2. After a couple of days, the occurrence is marked as having been symbolicated ('Symbolicated Crash Report'), but the symbolication has either not been applied to anything at all, or only to the iOS core calls.

Don't you have logs where you can check why this problem is occurring? I would say it should be fairly trivial to find out what is going on here.

Addendum: On-device symbolication has actually been activated in our most recent build and it has no effect on the problem.

[akornich]

@francois-iveo , I just looked at the item you shared here and that is what I found: Both dSYM bundles (the top ones on the list and dated as 2021-01-15) have zero-bytes size once unzipped and have no content (except empty directory structure). That would definitely cause symbolication failure.

[qnoppel]

@akornich: Thanks for that check! That explains a lot.

I looked at our setup for generating the dSYM zip and checked that it is identical to upload_dsym.py, so that can't be the problem.

I then dove into the build logs. Just the first run in a clean state with uploading disabled resulted in this:

That made me suspect a race condition or just some Xcode scheduling issue. I made the upload script the last build phase and tried repeated builds, which always ended up like this:

Then a clean build again:

And another clean build:

It definitely seems that there are cases where the dSYM file has not been generated (clean) or updated (repeated build) due to the order in which the build steps are executed. I think the first thing to do is to change upload_dsym.py to fail if the dSYM file does not exist or if its modification time isn't recent. Basic validation of the dSYM zips in the backend also seems like a pretty solid plan.

[akornich]

@qnoppel , thanks for the clarification! Yes, it does look like some conditions the Rollbar dSYM packaging and uploading script ran earlier than it is intended. However, I am not sure modifying the script will fix that. I know that fixing target build settings could be painful. A couple of years back, I gave up trying to analyze what was causing an odd behavior of a target for one of Xcode projects, nothing looked "wrong" with it. I ended up replacing it with a completely new build target definition (essentially implementing logically the same steps as the old one). And surprise - it did work fine. My guess is that the old target was defined years ago with a much older version of Xcode and then went through multiple major Xcode and corresponding project file upgrades. So, something did not go well with upgrades. Also, every time I make changes to a build target definition, I find it very helpful to close Xcode, clear the project/workspace Derived Data, open the project/workspace in Xcode again and start testing the change.
As far as adding dSYM package validation to Rollbar upload API, we will probably do it, but as of now, I can not promise any ETA for it.

[francois-iveo]

@akornich Modifying the script would prevent silent failures and thus lengthy time wasting bug tracking sessions like the ones your users have had here.

For us specifically I have just switched to manually uploading dSYMs.

[idsky2]

@qnoppel @akornich Looks like something similar was happening to me. The py script successfully uploaded a dSYM file but it was only about 1MB instead of 58MB. The real final dSYM file I retrieved from the archive, manually uploaded, worked correctly. Perhaps XCode thinks scripts won't need the symbol file and runs everything in parallel on multiple cores (I'm using M1 Pro).

[ghost]

Hey all, We released the Rollbar Apple SDK v3.0.0, which deprecates the Rollbar iOS SDK. The Apple SDK is a modernized approach to collecting errors from Apple devices and fixes many iOS SDK problems and pain points. Please, consider migrating over to the Apple SDK. You can find its documentation here. Don't hesitate to open an issue or contact our support if you encounter any difficulties.