Question about Current State of Breakpad for iOS?

[deathlyrage]

I'm currently using sentry-cocoa and running into this issue: https://github.com/getsentry/sentry-cocoa/issues/978

I'm wondering if sentry-native with breakpad might get around or solve this problem as it doesn't have these issues on other platforms. I know its experimental but I'd love to know what state its currently at so I can test it out.

Is it at a state where it can capture a native exception and report it, or a placeholder can compile but not do much state.

Sorry for not using bug report template, not really a bug report.

[Swatinem]

The problem with iOS and Android breakpad right now is that neither of them runs our integration test suite.

However I have integrated the iOS part into a swift application with a custom transport and was able to capture a native crash.

I would say feel free to experiment with this yourself and if it works then fine ;-)

[deathlyrage]

Just gave it a shot, mostly working. Having a really rough time trying to figure out how to set the deployment target, seems to be ignored no matter how I set it, anything obvious im doing wrong?

export CMAKE_XCODE_ATTRIBUTE_IPHONEOS_DEPLOYMENT_TARGET="14.1"
export IPHONEOS_DEPLOYMENT_TARGET=14.1				
cmake -B sentry-native-xcode -GXcode -DIPHONEOS_DEPLOYMENT_TARGET="14.1" -DCMAKE_SYSTEM_NAME=iOS -DSENTRY_BACKEND=breakpad -DBUILD_SHARED_LIBS=OFF
cmake --build sentry-native-xcode --target sentry --config RelWithDebInfo

Defaulting to the latest xcode one of 14.3 or w/e

[deathlyrage]

Also not really having any stack trace or symbols referenced. Attachments work however.

[Swatinem]

This does look like a capture_event, right? That does not automatically attach a stacktrace.

You would have to use sentry_event_value_add_stacktrace for that.

https://github.com/getsentry/sentry-native/blob/1b151d10fdb4b1fa2e4440cff4a436d6186fc862/include/sentry.h#L373-L380

The more interesting bit however would be to see if hard crashes are being captured correctly.

[deathlyrage]

Managed to get the stacktrace to work, can't get crashes to work. Suffering from 413 response codes, payload too large w/e. Even though my custom transport does zlib compression on these. I'm trying to figure out how to get them off my iOS device so I can either censor them and upload them and we can figure out why they are so big and hitting the sentry limits.

[Swatinem]

The limits imposed on envelopes should be quite sufficient actually: https://develop.sentry.dev/sdk/envelopes/#size-limits We do have some customers that hit those, but they usually deal with a large number of threads.

[deathlyrage]

Just gave it a shot, mostly working. Having a really rough time trying to figure out how to set the deployment target, seems to be ignored no matter how I set it, anything obvious im doing wrong?

export CMAKE_XCODE_ATTRIBUTE_IPHONEOS_DEPLOYMENT_TARGET="14.1"
export IPHONEOS_DEPLOYMENT_TARGET=14.1				
cmake -B sentry-native-xcode -GXcode -DIPHONEOS_DEPLOYMENT_TARGET="14.1" -DCMAKE_SYSTEM_NAME=iOS -DSENTRY_BACKEND=breakpad -DBUILD_SHARED_LIBS=OFF
cmake --build sentry-native-xcode --target sentry --config RelWithDebInfo

Defaulting to the latest xcode one of 14.3 or w/e

Any ideas on this one? Do we have to add something in the make file to allow setting of deployment targets?

[Swatinem]

You have to use MACOSX_DEPLOYMENT_TARGET environment var, as explained in https://cmake.org/cmake/help/latest/variable/CMAKE_OSX_DEPLOYMENT_TARGET.html

The docs say the attribute is names as MACOSX even if you are dealing with ios.

Also see https://stackoverflow.com/a/34208904 maybe.

[deathlyrage]

Everything is currently working perfectly except the termination handler / minidump crashes. I'm getting no log output about them and sentry doesnt seem to send them or mention them when it starts back up. Anything I can do to debug this? Is there a way to debug this while being attached to Xcode? Or does that cause side effects that make testing it difficult?

Iv checked and even tried sentry reinstall backend and still can't capture the crash.

[deathlyrage]

[2021.03.10-19.01.39:049][  0]AlderonLog: Sentry: Starting Init!
[2021.03.10-19.01.39:049][  0]AlderonLog: AlderonSentry: Set DSN SentryDev to https://[email protected]/5577093
[2021.03.10-19.01.39:049][  0]AlderonLog: AlderonSentry: Set Database Path to /var/mobile/Containers/Data/Application/4D325A6C-2200-43E1-9062-922CF5B326CB/Library/PathOfTitans/Saved/sentry-db
[2021.03.10-19.01.39:049][  0]AlderonLog: Sentry: Adding Attachment: /var/mobile/Containers/Data/Application/4D325A6C-2200-43E1-9062-922CF5B326CB/Library/PathOfTitans/Saved/Logs/PathOfTitans.log
[2021.03.10-19.01.39:049][  0]SentryLog: using database path "/private/var/mobile/Containers/Data/Application/4D325A6C-2200-43E1-9062-922CF5B326CB/Library/PathOfTitans/Saved/sentry-db"
[2021.03.10-19.01.39:050][  0]SentryLog: starting transport
[2021.03.10-19.01.39:050][  0]AlderonLog: SentryCustomTransport: DSN: https://[email protected]/5577093
[2021.03.10-19.01.39:050][  0]AlderonLog: SentryCustomTransport: PublicKey: REDACTED
[2021.03.10-19.01.39:050][  0]AlderonLog: SentryCustomTransport: ProjectId: 5577093
[2021.03.10-19.01.39:050][  0]AlderonLog: SentryCustomTransport: Envelope Endpoint: https://REDACTED.ingest.sentry.io/api/5577093/envelope/
[2021.03.10-19.01.39:050][  0]SentryLog: starting backend
[2021.03.10-19.01.39:050][  0]SentryLog: sending envelope
[2021.03.10-19.01.39:050][  0]SentryLog: serializing envelope into buffer
[2021.03.10-19.01.39:054][  0]AlderonLog: Sentry: Init Success.
[2021.03.10-19.01.39:055][  0]SentryLog: merging scope into event
[2021.03.10-19.01.39:060][  0]SentryLog: adding attachments to envelope
[2021.03.10-19.01.39:061][  0]SentryLog: sending envelope
[2021.03.10-19.01.39:061][  0]SentryLog: serializing envelope into buffer
[2021.03.10-19.01.39:071][  0]AlderonLog: SentryCustomTransport: Compression: Zlib: Uncompressed Size: 134585 Compressed Size: 32594
[2021.03.10-19.01.40:112][ 28]AlderonLog: IAlderonCommon: SentryEventCallback(200): {}
[2021.03.10-19.01.40:846][ 50]AlderonLog: IAlderonCommon: SentryEventCallback(200): {"id":"722aca57321b43246647ce9a2349e335"}
[2021.03.10-19.02.12:160][984]SentryLog: merging scope into event
[2021.03.10-19.02.12:161][984]SentryLog: adding attachments to envelope
[2021.03.10-19.02.12:161][984]SentryLog: sending envelope
[2021.03.10-19.02.12:161][984]SentryLog: serializing envelope into buffer
[2021.03.10-19.02.12:176][984]AlderonLog: SentryCustomTransport: Compression: Zlib: Uncompressed Size: 151931 Compressed Size: 35792
[2021.03.10-19.02.13:547][ 26]AlderonLog: IAlderonCommon: SentryEventCallback(200): {"id":"ab67443c21054e7c7a775ef946ccf3dd"}

Here is another example of custom events I send that work

[Swatinem]

Is there a way to debug this while being attached to Xcode? Or does that cause side effects that make testing it difficult?

When I was testing, the debugger would just break at the segfault, and I wouldn’t be able to get into the signal handler.

[2021.03.10-19.01.39:049][  0]AlderonLog: AlderonSentry: Set Database Path to /var/mobile/Containers/Data/Application/4D325A6C-2200-43E1-9062-922CF5B326CB/Library/PathOfTitans/Saved/sentry-db
[2021.03.10-19.01.39:049][  0]SentryLog: using database path "/private/var/mobile/Containers/Data/Application/4D325A6C-2200-43E1-9062-922CF5B326CB/Library/PathOfTitans/Saved/sentry-db"

Is that an iOS thing that it adds another /private/ prefix in that case?

What kind of crash are you triggering? I was able to get a minidump for an explicit segfault that I triggered.

However all my testing was done in the simulator. I don’t have any real device to test that stuff with.

[deathlyrage]

• Is that an iOS thing that it adds another /private/ prefix in that case? Yes, I call a method to convert it to a system friendly path from a unreal engine internal sandbox path else sentry init fails because it can't write to the directory

• Type of Crash, just currently doing a deref / function call on a nullptr.

• Testing done on real device, my game can't run in the simulator.

[deathlyrage]

Interesting discovery using the same code for Android results in a minidump and event being sent and created, so its specifically the crash termination handler for iOS breakpad.

[yuta-ishii-ts]

Even with the latest code, I can't seem to get notified of crashes when trying to incorporate sentry-native into iOS. Perhaps it does not work well with sandboxed apps.

I am developing a cross-platform app that is developed in C++ and would be very happy to be able to use sentry-native. Do you have plans to resolve this issue in the near future? Or do we have to use sentry-cocoa for iOS apps?

[supervacuus]

Hi, @yuta-ishii-ts! AFAIK the current iOS support in the Native SDK is considered experimental.

What does this mean concretely:

• the current goto SDK for anything *OS is sentry-cocoa
• iOS support for the breakpad backend does not include a mechanism for uploading the respective report, so you'll have to upload the generated envelope from the app.
• Many other cross-platform developers want cross-platform support in the Native SDK rather than incorporating two or more SDKs into their code base. But right now, this is not a priority.

[StasDzundzaGL]

Hi @yuta-ishii-ts! I had a similar issue to yours. I was also struggling to integrate sentry-native into my cross-platform C++ app, and fortunately, I managed to do this today :). Here are the instructions:

1. Build CURL for iOS. I used vcpkg for this task: ./vcpkg install curl --triplet arm64-ios
2. Build sentry-native with curl and breakpad:

cmake -B ./build -GXcode -DSENTRY_BACKEND=breakpad -DCMAKE_OSX_DEPLOYMENT_TARGET=13.1 -DSENTRY_BUILD_TESTS=OFF -DSENTRY_BUILD_EXAMPLES=OFF -DSENTRY_BUILD_SHARED_LIBS=OFF -DSENTRY_TRANSPORT=curl -DCURL_INCLUDE_DIR=/Users/your.user/path/to/cURL/include -DCURL_LIBRARY=/Users/your.user/path/to/cURL/lib/libcurl.a -DCMAKE_INSTALL_PREFIX="../sentry-native-ios" -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_SYSTEM_NAME=iOS

cmake --build ./build --config RelWithDebInfo --target install --parallel

3. Integrate the build lib into your build system
4. Upload generated .dSYM files to Sentry
5. Test it

PS: I tried to use inproc backend, but it didn't send stack trace to Sentry when app was crashing.

[yuta-ishii-ts]

@supervacuus Thanks for your response! It was about what I expected, but it was helpful to have it clearly explained.

@StasDzundzaGL Thanks for sharing your information! I'll give it a try later.

[supervacuus]

Thanks for testing the curl-transport on iOS, @StasDzundzaGL. I guess there was a reason why it was deliberately disabled for the iOS build, but it might no longer hold. @Swatinem, can you remember why the iOS build had no transport by default?

[yuta-ishii-ts]

@supervacuus @Swatinem Do you plan to support curl-transport on iOS in the near future? I'm not sure whether to wait until it is officially supported or patch it independently, so I'd appreciate it if you could let me know as a reference.

[supervacuus]

I am not sure what you mean @yuta-ishii-ts.

According to https://github.com/getsentry/sentry-native/issues/493#issuecomment-2032678178, it seems like there is no need to patch anything. You only need to tell the build to use the curl transport during the configuration phase. What parts would you patch?

[yuta-ishii-ts]

@supervacuus Since macOS does not require the curl build to be done by myself, I assumed that the curl build would be performed in the sentry build command. But this seems to be a mistake, sorry.

[supervacuus]

Thanks @yuta-ishii-ts! Then let me clarify a bit:

• we typically do not add transports to our mobile builds because packaging curl in a mobile artifact is often not desired due to size (and because there is typically an HTTP client library available on the platform or in the host programming environment).
• On Android, the downstream SDK (sentry-android) takes over the responsibility for the transport.
• On iOS, there is no downstream SDK because sentry-cocoa uses another native crash backend.
• If there is ever full iOS support in sentry-native, we would probably add another transport in terms of NSURLConnection/NSURLSession, but there are no plans to do this at the moment.

If you consider the configuration phase of the build mentioned in https://github.com/getsentry/sentry-native/issues/493#issuecomment-2032678178, only these two are iOS-specific

-DCMAKE_OSX_DEPLOYMENT_TARGET=13.1 
-DCMAKE_SYSTEM_NAME=iOS

Only these are relevant to the curl setup:

-DSENTRY_TRANSPORT=curl 
-DCURL_INCLUDE_DIR=/Users/your.user/path/to/cURL/include 
-DCURL_LIBRARY=/Users/your.user/path/to/cURL/lib/libcurl.a 

If you define SENTRY_TRANSPORT=curl, this only means that we find_package() libcurl. So specifying CURL_INCLUDE_DIR and CURL_LIBRARY is only necessary if the configuration environment doesn't allow FindCURL.cmake to find the library (for instance, because you installed libcurl via vcpkg or built it from source, and thus, artifacts ended up in non-standard paths).