Support for `--obfuscate` and `--split-debug-info` for iOS/macOS apps

[bruno-garcia]

This issue is to track the support for --obfuscate and --split-debug-info on iOS apps.

The original request was done on the flutter/flutter issue tracker and since then, Android support was added to Sentry.

For this to happen, Sentry expect symbols on iOS to be in a 'standard' format for that platform. Namely Mach-O files. The request for the Dart team for this support is tracked here.

Support for desktop is tracked separately, here.

[t-kozak]

Hi Folks from Sentry and Dart teams,

This problem is in a bit of a hot potato state...

The Sentry team:

"expects symbols on iOS to be in a 'standard' format for that platform. Namely Mach-O files."

while the Dart team:

We are not actively working on this at the moment, because high priority uses cases are all covered to the best of our knowledge.

I want to obfuscate my flutter code in release build, and I definitely want to see error stack trace

You can still manually symbolicate and deobfuscate your crashes, we emit enough information for you to do it.

It is fair to assume that due to this state, the problem will not be resolved anytime soon, unless some team opens up for a compromise.

Since the debugging symbols produced for iOS are compatible with the Linux binary format (assuming I understood the various conversations correctly), would it be possible for Sentry to allow us to mark given crash as coming from Flutter, and de-symbolicate reports like these using the android toolchain, regardless of platform (iOS or Android)?

Alternatively, if I'm understanding this wrong, would you reconsider using the flutter symbolise tool in the interim? Or adding a webhook or something that allows us to integrate with any pre- or post- processing flow?

All in all, it's getting difficult to treat flutter as a platform for "large scale", not-a-proof-of-concept products. You cannot expect developers to manually de-symbolicate every crash - it scales terribly. Also, due to the size of the runtime, removal of debugging symbols and obfuscation is really a must rather than nice to have.

Thanks, Ted

[bruno-garcia]

@jan-auer can we shell out to flutter symbolize? This is gradually building up to be a big blocker.

[mraleph]

I followed up on dart-lang/sdk#43612

tldr; is that Flutter CLI can be changed to produce necessary dSYMs. No changes in Dart compilation toolchain are required. I have provided the prototype of necessary changes - just looking for somebody to take it over the finishing line, which means clean it up, fix flutter symbolize (which requires contributing some refactoring to package native_stack_traces), add tests per Flutter contribution guidelines, get PR reviewed and landed).

[marandaneto]

@mraleph thanks a lot for your feedback and prototype, we'll take a look into it.

[fzyzcjy]

Hi, is there any updates? This is quite important and commonly used! Thanks!

[t-kozak]

@marandaneto Have you guys had a chance to look into @mraleph 's prototype?

Can you clarify Dart's intent here? Do you plan to incorporate this changes or is it shelved and ghosted in a hope that folks stop complaining? :)

[marandaneto]

it's part of our Q4 (which ends at the end of January) plan but we're not there yet.

We'll need some guidance from the Flutter/Dart team since all the patches are in there and not in sentry itself.

@t-kozak sentry, dart, and flutter are also open-source, feel free to open PRs as well, you can help us out on that if you have some urgency.

[t-kozak]

Hey @marandaneto thanks for the heads up. Q4 sounds good.

I honestly thought this is more on Flutter side than yours. Thanks for your hard work!

[marandaneto]

@t-kozak it is on the Dart and Flutter side of things, but we'll try to submit patches as suggested here https://github.com/getsentry/sentry-dart/issues/444#issuecomment-842169869

[marandaneto]

Related either for better Android or iOS support. https://github.com/dart-lang/sdk/issues/44325 https://github.com/dart-lang/sdk/issues/44350 https://github.com/dart-lang/sdk/issues/43274

[fzyzcjy]

Looking forward to updates!

[marandaneto]

Flutter PR https://github.com/flutter/flutter/pull/101586 and Dart PR https://github.com/dart-lang/sdk/pull/48767

[bruno-garcia]

https://dart-review.googlesource.com/c/sdk/+/242108

[vaind]

The latest PR is https://dart-review.googlesource.com/c/sdk/+/250381

[scott-the-brewer]

And another one: https://dart-review.googlesource.com/c/sdk/+/251464

[shripal-lean-ix]

I see that the PRs were merged? So are we now able to upload debug files for flutter builds with --obfuscate and --split-debug-info for iOS apps? If yes, can you please mention the steps for the same?

[vaind]

I see that the PRs were merged? So are we now able to upload debug files for flutter builds with --obfuscate and --split-debug-info for iOS apps? If yes, can you please mention the steps for the same?

No, that's far from done. Only the PR to get the needed functionality to the Dart SDK is merged now. There still need to be changes in Flutter (https://github.com/flutter/flutter/pull/101586) which will also need to come out in a release after they're done and merged, of course, then there's a PR in the sentry-dart SDK (https://github.com/getsentry/sentry-dart/pull/823).

[shripal-lean-ix]

Thanks a lot for the clarification. Actively following the progress on the issue.

[bruno-garcia]

There's seem to be some movement from the Dart team too: https://github.com/dart-lang/sdk/commit/24683da915867609b9b52bd2124e6fb603b9f160

[vaind]

I've added missing tests in the Flutter PR and then tried macOS which shares some of the code and it didn't work. Tess Strickland is making changes in the dart SDK to support parsing multi-architecture dSYM mach-o & symbolicate stack traces which also missed the architecture info. After that's done, the Flutter PR is hopefully not far from completion.

[vaind]

The Flutter tool PR is being merged now so unless something prevents that, the dSYM generation be available soon on the master channel. I'm updating the sentry-dart PR so we'll be ready to work with that.

[marandaneto]

Thanks for the update @vaind

[LeoRpc]

Hi guy,

I cannot make it works guys :'( But I tried my best: reading the docs, reading github issues, waiting ton of time for my poor 2015 macbook pro to build my flutter app to make tests... Soooo painful! Pleeeease help me :P

1- Inaccuracies I saw on the documentation

1. Package documentation: "Known limitations, Flutter split-debug-info and obfuscate flags aren't supported on iOS yet, but only on Android, if this feature is enabled, Dart stack traces are not human readable"
2. Sentry docs for Flutter: "Flutter split-debug-info and obfuscate flags are supported on iOS/macOS."
3. Sentry docs for Flutter: "... Learn how to upload the dSYM files" send to the native apple documentation, not the flutter documentation. The corresponding flutter documentation also point on the same native apple documentation. This could be OK, BUT, as far as I know, the flutter build ipa/ios --obfuscate --split-debug-info=$SPLIT_DEBUG_PATH does not outputs dSYM files (as mentionned in the native apple documentation), but .symbols files which adds to the confusion!

2- My issue

Note

1. It works well for Android, but not for iOS.
2. I come from web development so I have no background on flutter or native development.
3. I use Flutter 3.7.2, sentry_flutter: ^6.19.0, flutter doctor all ok...

Issue

1. flutter build appbundle --obfuscate --split-debug-info=$SPLIT_DEBUG_PATH
2. sentry-cli difutil check $SPLIT_DEBUG_PATH/app.android-x64.symbols (one random android .symbol file) outputs: `Type: elf debug companion Contained debug identifiers:

   Debug ID: 8e504493-5a69-af67-4c94-29828efc4ec9 Code ID: 9344508e695a67af4c9429828efc4ec9 Arch: x86_64 Contained debug information: symtab, debug Usable: yes`

3. flutter build ipa --obfuscate --split-debug-info=$SPLIT_DEBUG_PATH
4. sentry-cli difutil check $SPLIT_DEBUG_PATH/app.ios-arm64.symbols (one random ios .symbol file) outputs: `Type: elf debug companion Contained debug identifiers:

   Debug ID: 00000000-0000-0000-0000-000000000000 Arch: arm64 Contained debug information: symtab, debug Usable: no (missing debug identifier, likely stripped)`

5. flutter build **ios** --obfuscate --split-debug-info=$SPLIT_DEBUG_PATH -> same result

Question

Is there an issue with the way I build my flutter app (maybe the expected split-debug-info files are really dSYM files?)? With Sentry? Sentry-cli? Directions to investigate?

Thanks in advance! <3 Leo