split-debug-info doesn't "symbolize"

[sergiy-sc]

Platform

Dart

Obfuscation

Disabled

Debug Info

Enabled

Doctor

[!] Flutter (Channel [user-branch], 3.13.9, on macOS 14.1.2 23B92 darwin-arm64, locale en-FI) ! Flutter version 3.13.9 on channel [user-branch] at /Downloads/flutter Currently on an unknown channel. Run flutter channel to switch to an official channel. If that doesn't fix the issue, reinstall Flutter by following instructions at https://flutter.dev/docs/get-started/install. ! Upstream repository unknown source is not a standard remote. Set environment variable "FLUTTER_GIT_URL" to unknown source to dismiss this error. • Framework revision d211f42860 (4 months ago), 2023-10-25 13:42:25 -0700 • Engine revision 0545f8705d • Dart version 3.1.5 • DevTools version 2.25.0 • If those were intentional, you can disregard the above warnings; however it is recommended to use "git" directly to perform update checks and upgrades.

[!] Android toolchain - develop for Android devices (Android SDK version 34.0.0) • Android SDK at /Users/../Library/Android/sdk ✗ cmdline-tools component is missing Run path/to/sdkmanager --install "cmdline-tools;latest" See https://developer.android.com/studio/command-line for more details. ✗ Android license status unknown. Run flutter doctor --android-licenses to accept the SDK licenses. See https://flutter.dev/docs/get-started/install/macos#android-setup for more details.

Version

7.14.0

Steps to Reproduce

1. build with flutter build apk --release --split-debug-info=test_app_debug_info
2. upload symbols with flutter packages pub run sentry_dart_plugin
3. Throw an exception

Expected Result

Symbolized stack trace

Actual Result

Sentry shows:

I checked quite carefully release number and package name and the match. However I have a feeling that sentry_dart didn't upload stack trace correctly, json has this:

 "exception": {
    "values": [
      {
        "type": "_Exception",
        "value": "Exception: Test exception",
        "stacktrace": {
          "frames": [
            {
              "platform": "native",
              "in_app": false,
              "data": {
                "orig_in_app": -1,
                "symbolicator_status": "unknown_image"
              },
              "instruction_addr": "0x7650cdaff7"
            },
            {
              "platform": "native",
              "in_app": false,
              "data": {
                "orig_in_app": -1,
                "symbolicator_status": "unknown_image"
              },
              "instruction_addr": "0x7650cd9983"
            },
            {
              "platform": "native",
              "in_app": false,
              "data": {
                "orig_in_app": -1,
                "symbolicator_status": "unknown_image"
              },
              "instruction_addr": "0x7650cd99ff"
            },

and if I try to symbolize "raw stacktrace" which I get from web:

_Exception: Exception: Test exception
  0x76510374d3 null
  0x7650f7de4f null
  0x7650f7dd4b null
  0x765137431f null

I don't get anything. I can see that this is wrong format. If I print exception in my app, I get something like this:

 #00 abs 00000076511756ff virt 00000000002756ff _kDartIsolateSnapshotInstructions+0x1be2bf
 #01 abs 0000007651081917 virt 0000000000181917 _kDartIsolateSnapshotInstructions+0xca4d7
 #02 abs 00000076510816d3 virt 00000000001816d3 _kDartIsolateSnapshotInstructions+0xca293
 #03 abs 000000765117846f virt 000000000027846f _kDartIsolateSnapshotInstructions+0x1c102f
 #04 abs 000000765112fadf virt 000000000022fadf _kDartIsolateSnapshotInstructions+0x17869f
 #05 abs 00000076511196cb virt 00000000002196cb _kDartIsolateSnapshotInstructions+0x16228b
 #06 abs 00000076511248d3 virt 00000000002248d3 _kDartIsolateSnapshotInstructions+0x16d493

which transforms into:

#0      _MyHomePageState.build.<anonymous closure> (/Users/Projects/work/zoo/calculator/lib/main.dart:84:21)
#1      _InkResponseState.handleTap (/Users/Downloads/flutter/packages/flutter/lib/src/material/ink_well.dart:1154:21)
#2      _InkResponseState.handleTap (/Users/Downloads/flutter/packages/flutter/lib/src/material/ink_well.dart:1146:3)
#3      GestureRecognizer.invokeCallback (/Users/Downloads/flutter/packages/flutter/lib/src/gestures/recognizer.dart:275:24)
#4      TapGestureRecognizer.handleTapUp (/Users/Downloads/flutter/packages/flutter/lib/src/gestures/tap.dart:654:11)
#5      BaseTapGestureRecognizer._checkUp (/Users/Downloads/flutter/packages/flutter/lib/src/gestures/tap.dart:311:5)

any ideas what I do wrong?

Are you willing to submit a PR?

None

[buenaflor]

Hey can you tell us a bit more about your set up? Is this only affecting android or also other platforms?

[sergiy-sc]

I tried it only on Android. We are using sentry_dart in a Flutter app, can't use flutter sentry because it conflicts with C++ host which also uses Sentry.

[sergiy-sc]

I tried to create a new project and use sentry_flutter and it worked flawlessly. However when I replace sentry_flutter with sentry I couldn't get it to work. If I understand correctly, sentry server can't match exception with binary because of wrong release string. But I'm quite sure it's correct. May be it also wants correct OS string? Here is a repo, https://github.com/sergiy-sc/calculator (need to update dsn, and tweak yml to match org). Doc is a bit frustrating https://docs.sentry.io/platforms/flutter/upload-debug/ because it doesn't explain how to create proper auth_token and how to make it work with url. I guess I'm creating auth_token in wrong place because upload tool complains that project url doesn't match url from token. I guess it's not that important, because sentry_flutter works but still...

[buenaflor]

The information regarding the auth token can be found here

For more clarification on your stack, you use sentry (dart) on your flutter app which underneath uses our sentry native sdk?

[sergiy-sc]

The information regarding the auth token can be found here

Yes I read that document, but which one to use with debug info? Organization token? It would be easier if debug info doc specified what is the best token, and which scopes/permissions need to be enabled (if any). Frankly I don't understand why I should use organization token to upload symbols for a specific project. And again, if I use organization token, what should be the project url so that they both work?

For more clarification on your stack, you use sentry (dart) on your flutter app which underneath uses our sentry native SDK?

Let's try to get it working with a simple Calculator (https://github.com/sergiy-sc/calculator) app which uploaded to github. I think if we can get it working, then I will be able to modify our real app config. Real app is a C++ app (uses sentry natively), that embeds Flutter Engine (which uses sentry via dart only package).

[buenaflor]

The scope and permissions need to be better documented, I agree.

I guess I'm creating auth_token in wrong place because upload tool complains that project url doesn't match url from token.

What kind of token did you use? Organization?

[sergiy-sc]

The token issue is not important for the original issue. Let's put it aside for a moment. Let's try to make Sentry server find correct debug symbol files. They are uploaded for sure, I can see them in a corresponding page, but server doesn't match them for dart project.

Let's try to get it working with a simple Calculator (https://github.com/sergiy-sc/calculator) app which uploaded to github. I think if we can get it working, then I will be able to modify our real app config. Real app is a C++ app (uses sentry natively), that embeds Flutter Engine (which uses sentry via dart only package).

[kahest]

Hey @sergiy-sc thanks for the details, we need to replicate your setup and investigate. @buenaflor is currently OOO, so this may take a bit longer, but we'll get back to you here.

[denrase]

@kahest I checked out the sample project and could reproduce everything as described. I think someone with more knowledge on symbolication should check this issue. Linke below are the two issues in my project setup. With the dart only setup, we get "symbolicator_status": "unknown_image" in the frames. I also attached the raw events for both flutter & dart.

sentry-flutter

https://denrase.sentry.io/issues/5058823345/?project=4506898490589184&query=is%3Aunresolved&referrer=issue-stream&statsPeriod=14d&stream_index=2

sentry

https://denrase.sentry.io/issues/5058676430/?project=4506898490589184&query=is%3Aunresolved&referrer=issue-stream&statsPeriod=14d&stream_index=0

calculator_event_sentry.json calculator_event_sentry_flutter.json

[denrase]

@buenaflor @kahest @stefanosiano Just checked the LoadImageList integration. We do depend on the native iOS/Android SDKs here, as we fetch debug images over the method channel.

https://github.com/getsentry/sentry-dart/blob/e82709a24504959ed7fe9b79ba3f038a31b6c5db/flutter/lib/src/integrations/load_image_list_integration.dart#L62

Ran the sample application with SentryFlutter and removed the integration. To me it looks like the behaviour then is the same as with the dart only SDK, so this might indeed be the root of the issue.

Flutter SDK

https://denrase.sentry.io/issues/5058823345/?project=4506898490589184&query=is%3Aunresolved&referrer=issue-stream&statsPeriod=14d&stream_index=0

Flutter SDK without LoadImageListIntegration integration

https://denrase.sentry.io/issues/5058676430/events/e7d9fc058b364491b1e1bcf0425f55bf/?project=4506898490589184&query=is%3Aunresolved&referrer=next-event&statsPeriod=14d&stream_index=0

[denrase]

@sergiy-sc Is it possible for you to just use the Sentry Flutter SDK? If not, what are the issues that you are facing in your setup?

[sergiy-sc]

@sergiy-sc Is it possible for you to just use the Sentry Flutter SDK? If not, what are the issues that you are facing in your setup?

No, because the game that embeds flutter engine uses its own sentry, and if I use flutter sentry it crashes. I think that's because of version mismatch. Also, we don't need any of the stuff flutter-sentry adds on top, so it would be a waste of space. It's not possible to get it working with dart only solution?

[denrase]

@sergiy-sc thx for the info, we will discuss how to best resolve this \cc @buenaflor @kahest

[vaind]

@denrase your findings are correct, we're relying on the list of native images that is being provided by native Android/iOS implementation. I don't see how this could reasonably move directly to sentry-dart package.

[vaind]

Also, we don't need any of the stuff flutter-sentry adds on top, so it would be a waste of space. It's not possible to get it working with dart only solution?

Sentry for Flutter doesn't only add features for Flutter directly, but also platform-specific error handling. This applies to the dart errors you're having problem with but it also adds support for native crashes.

No, because the game that embeds flutter engine uses its own sentry, and if I use flutter sentry it crashes. I think that's because of version mismatch.

This comes up occasionally when a library tries to integrate sentry. The general recommendation is that it shouldn't be done - you're seeing one of the issues with that. Another one could be that error capture mechanism don't have a sure way to figure out which instance should handle that error. It could try to figure it out based on stacks, but that's is certain to be wrong in some cases. See the highlighted bit here for more details: https://docs.sentry.io/platforms/

My suggestion would be to remove Sentry from the underlying library, if you have control over it, or ask the developer to do so, if you don't.

[denrase]

@vaind Thank you for providing more context and recommentations! 🙇