Include source maps for staging apps

[roryabraham]

If you haven’t already, check out our contributing guidelines for onboarding and email [email protected] to request to join our Slack channel!

---

Problem

We are starting to automatically create GitHub issues from mobile crash reports produced by Firebase Crashlytics for the staging and production mobile apps. These crash reports include stacktraces that are difficult or impossible to follow, because they reference lines in a bundled application, not lines in the source code.

Solution

Generate and include source maps for the staging apps so that stacktraces can reference lines in source code rather than lines from a massive minified bundle. Investigate the drawbacks of including these source maps in the staging app, and consider whether we could also include them in the production app.

View all open jobs on GitHub

[roryabraham]

cc @AndrewGable

[roryabraham]

Full-transparency: I really don't know much about the topic of source maps, how they work, or their benefits and drawbacks. But I believe having better stacktraces in crash reports would help us to more quickly and successfully trace and fix crashes.

[roryabraham]

cc @kidroca too, since this seems like something you might be knowledgable about

[kidroca]

I think since the code is already open source we should be fine including source maps along the bundled code

There should be a way to tell crashlytics to use source maps. I can work on this and investigate what's needed

---

Full-transparency: I really don't know much about the topic of source maps, how they work, or their benefits and drawbacks.

No one does 😁

Benefits

• seeing real source while debugging
• refference real lines in stack traces

Drawbacks

• people can easily see your source

[AndrewGable]

Agree no concern because it's open source, this would be great to look into.

[roryabraham]

It might be tough for @kidroca to look into the crashlytics piece, because he won't have access to Firebase.

[AndrewGable]

Once we get the source maps in place, I'd be happy to check Firebase

[kidroca]

web and desktop already include source maps you can find them this way:

1. find bundle name
2. append .map to it

Here's the current one for staging: https://staging.new.expensify.com/app-cc32624656c52b1a56be.bundle.js.map

• Select the ...bundle.js file, then "Open in new tab", then append .map to url

[roryabraham]

Nice! Crashlytics is only relevant to iOS/Android for now. Do you know if those apps ship with a source map?

[kidroca]

Source maps for iOS/Android should already be generated, the problem is even with source maps Firebase does not accept uploading a source map file, though stack traces can be decoded manually: https://github.com/invertase/react-native-firebase/issues/2327#issuecomment-847477220

So instead of trying to make Firebase work with source maps (show mapped sources) we can decode the stack trace and create a github issue mapped to original source

If sending/using source maps was possible it would have been explained here: https://rnfirebase.io/crashlytics/usage

Generating source maps

As mentioned they should already be generated, but since creating an issue will happen in a different context it should

1. When an error happens - extract error, stack trace and app version
2. Use app version so that we can map to the given version source
   • Update release scripts so they capture source-maps as artifacts. E.g. capture v1.1.77-0.ios/android/web.source.js.map in job artifacts to allow a future pipeline to use them
   • Alternatively the job should checkout the repository at the given version and generate source maps each time

To generate source maps for iOS/Android we can run the following commands:

iOS

react-native bundle \
  --dev false \
  --platform ios \
  --entry-file index.ios.js \
  --bundle-output main.jsbundle \
  --sourcemap-output main.jsbundle.map

Android

react-native bundle \
  --dev false \
  --platform android \
  --entry-file index.android.js \
  --bundle-output index.android.bundle \
  --sourcemap-output index.android.bundle.map

The build process probably already generates these so we should just find where and expose them as build artifacts

Symbolicating - decoding minified stack trace to original source

Covered in: https://reactnative.dev/docs/symbolication

Given a piece of text containing minified stack trace (we'll be getting this from Firebase somehow) We run

npx metro-symbolicate android/app/build/generated/sourcemaps/react/release/index.android.bundle.map < stacktrace.txt

to convert it to original source that we can use to create the Github issue

---

I think Firebase does not support js source maps (in iOS/Android), because in the Firebase Console we have Web/iOS/Android apps and each source is mapped to the underlying architecture, so iOS sources are mapped to iOS code (which is mapped at compile time) and the service knows how to decode that, while with js sources it would be up to react-native-firebase to remap js stack traces before posting crashlytics to Firebase and I guess that's an extra step they don't want to do for various reasons (just my thoughts though, not a fact)

[kidroca]

Feature request for react-native-firebase to use source maps for Crashlytics: https://invertase.canny.io/react-native-firebase/p/crashlytics-to-take-in-javascript-soucemap

[AndrewGable]

@kidroca - Here are example issues that I am creating using real Firebase data: https://github.com/Andrew-Test-Org/Public-Test-Repo/issues (they will be added into App soon)

Can you confirm that these have the data we need to run the npx metro-symbolicate command?

[kidroca]

Looks like Android sources have more usable stack traces than iOS Sample ticket: https://github.com/Andrew-Test-Org/Public-Test-Repo/issues/276

See how iOS has no line:column numbers only function names

Ticket top information - main.jsbundle:196451:30 line 196451 shouldComponentUpdate seem to come from iOS - so there should be line information available somewhere

---

Symbolicating is actually a bit more involved than bundling with the above commands, besides the original documentation info is scarcely available, I followed 2 guides in order to produce the following results

Decoded stacks from Andrew-Test-Org/Public-Test-Repo#276 Android

A familiar keyChanged is indeed linked to Onyx.js

.shouldComponentUpdate (address at src/pages/home/report/ReportActionsView.js:125:AppState.addEventListener$argument_1:)
.checkShouldComponentUpdate (address at node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:2527:ch
eckShouldComponentUpdate:)
.updateClassComponent (address at node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:4460:updateCl
assComponent:)
.beginWork (address at node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:7354:beginWork$1:)
.performUnitOfWork (address at node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:6771:performUnit
OfWork:)
.workLoopSync (address at node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:6764:workLoopSync:)
.renderRootSync (address at node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:6746:renderRootSync
:)
.performSyncWorkOnRoot (address at node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:6454:perform
SyncWorkOnRoot:)
.flushSyncCallbacks (address at node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:2058:flushSyncC
allbacks:)
.scheduleUpdateOnFiber (address at node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:6211:schedul
eUpdateOnFiber:)
.enqueueSetState (address at node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:2489:classComponen
tUpdater.enqueueSetState:)
.anonymous (address at node_modules/react/cjs/react.production.min.js:12:C.prototype.setState:)
.anonymous (address at node_modules/react-native-onyx/lib/Onyx.js:339:_.each$argument_1:)
.each (address at node_modules/underscore/underscore-umd.js:1335:each:)
.keyChanged (address at node_modules/react-native-onyx/lib/Onyx.js:319:keyChanged:)
.anonymous (address at node_modules/react-native-onyx/lib/Onyx.js:715:_.each$argument_1:)
.each (address at node_modules/underscore/underscore-umd.js:1330:each:)
.anonymous (address at node_modules/react-native-onyx/lib/Onyx.js:714:getAllKeys.then$argument_0:)
.tryCallOne (null:null:null:)
.anonymous (null:null:null:)
.anonymous (address at node_modules/react-native/Libraries/Core/Timers/JSTimers.js:248:_allocateCallback$argument_0:)
._callTimer (address at node_modules/react-native/Libraries/Core/Timers/JSTimers.js:112:_callTimer:)
._callReactNativeMicrotasksPass (address at node_modules/react-native/Libraries/Core/Timers/JSTimers.js:166:_callReactNativeMicrotasksPa
ss:)
.callReactNativeMicrotasks (address at node_modules/react-native/Libraries/Core/Timers/JSTimers.js:418:callReactNativeMicrotasks:)
.__callReactNativeMicrotasks (address at node_modules/react-native/Libraries/BatchedBridge/MessageQueue.js:393:__callReactNativeMicrotas
ks:)
.anonymous (address at node_modules/react-native/Libraries/BatchedBridge/MessageQueue.js:135:__guard$argument_0:)
.__guard (address at node_modules/react-native/Libraries/BatchedBridge/MessageQueue.js:370:__guard:)
.flushedQueue (address at node_modules/react-native/Libraries/BatchedBridge/MessageQueue.js:134:flushedQueue:)
.invokeCallbackAndReturnFlushedQueue (address at node_modules/react-native/Libraries/BatchedBridge/MessageQueue.js:130:invokeCallbackAnd
ReturnFlushedQueue:)

I've checked out v 1.1.70-1 and ran gradlew assembleRelease to generate the release source maps and then use `metro-symbolicate to remap the stack trace

Help materials:

• The guide here explains how we can build regular js then hermes then combine source maps: https://docs.sentry.io/platforms/react-native/manual-setup/hermes/#compile-sourcemaps
• While this one seems slightly easier: https://docs.bugsnag.com/build-integrations/js/source-maps-react-native/#hermes
  • ./gradlew assembleRelease indeed builds everything needed for Android We can use the .map file generated in ./android

    metro-symbolicate android/app/build/generated/sourcemaps/react/release/index.android.bundle.map < stacktrace.txt
    

  • while on iOS we need to tweak a Build Phase in order to produce source maps

[AndrewGable]

I might be able to improve iOS, can you try with this data? I pulled it directly from Firebase UI:

Non-fatal Exception: JavaScriptError
0  ???                            0x0 shouldComponentUpdate + 199284 (main.jsbundle:199284:30:199284)
1  ???                            0x0 checkShouldComponentUpdate + 5581 (main.jsbundle:5581:109:5581)
2  ???                            0x0 updateClassComponent + 6842 (main.jsbundle:6842:382:6842)
3  ???                            0x0 beginWork$1 + 8980 (main.jsbundle:8980:226:8980)
4  ???                            0x0 performUnitOfWork + 8525 (main.jsbundle:8525:27:8525)
5  ???                            0x0 workLoopSync + 8514 (main.jsbundle:8514:24:8514)
6  ???                            0x0 renderRootSync + 8496 (main.jsbundle:8496:21:8496)
7  ???                            0x0 performSyncWorkOnRoot + 8233 (main.jsbundle:8233:36:8233)
8  ???                            0x0 flushSyncCallbacks + 5165 (main.jsbundle:5165:32:5165)
9  ???                            0x0 scheduleUpdateOnFiber + 8047 (main.jsbundle:8047:345:8047)
10 ???                            0x0 enqueueSetState + 5551 (main.jsbundle:5551:38:5551)
11 ???                            0x0 anonymous + 73232 (main.jsbundle:73232:33:73232)
12 ???                            0x0 anonymous + 63722 (main.jsbundle:63722:47:63722)
13 ???                            0x0 each + 61811 (main.jsbundle:61811:19:61811)
14 ???                            0x0 keyChanged + 63704 (main.jsbundle:63704:29:63704)
15 ???                            0x0 anonymous + 63943 (main.jsbundle:63943:19:63943)
16 ???                            0x0 each + 61805 (main.jsbundle:61805:19:61805)
17 ???                            0x0 anonymous + 63942 (main.jsbundle:63942:31:63942)
18 ???                            0x0 tryCallOne + 53 (InternalBytecode.js:53:16:53)
19 ???                            0x0 anonymous + 139 (InternalBytecode.js:139:27:139)
20 ???                            0x0 apply ((native):0:0)
21 ???                            0x0 anonymous + 11428 (main.jsbundle:11428:26:11428)
22 ???                            0x0 _callTimer + 11336 (main.jsbundle:11336:17:11336)
23 ???                            0x0 _callReactNativeMicrotasksPass + 11367 (main.jsbundle:11367:17:11367)
24 ???                            0x0 callReactNativeMicrotasks + 11573 (main.jsbundle:11573:44:11573)
25 ???                            0x0 __callReactNativeMicrotasks + 3189 (main.jsbundle:3189:46:3189)
26 ???                            0x0 anonymous + 3064 (main.jsbundle:3064:45:3064)
27 ???                            0x0 __guard + 3172 (main.jsbundle:3172:15:3172)
28 ???                            0x0 flushedQueue + 3063 (main.jsbundle:3063:21:3063)
29 ???                            0x0 invokeCallbackAndReturnFlushedQueue + 3056 (main.jsbundle:3056:33:3056)

[kidroca]

I might be able to improve iOS, can you try with this data? I pulled it directly from Firebase UI:

Can you also share App version from which the stack traces are coming from? I was able to produce some mappings using 1.1.70-1 but they don't align with the original source

[AndrewGable]

Attaching a new one since I am not 100% confident that this is the same exact one:

Non-fatal Exception: JavaScriptError
0  ???                            0x0 shouldComponentUpdate + 199284 (main.jsbundle:199284:30:199284)
1  ???                            0x0 checkShouldComponentUpdate + 5581 (main.jsbundle:5581:109:5581)
2  ???                            0x0 updateClassComponent + 6842 (main.jsbundle:6842:382:6842)
3  ???                            0x0 beginWork$1 + 8980 (main.jsbundle:8980:226:8980)
4  ???                            0x0 performUnitOfWork + 8525 (main.jsbundle:8525:27:8525)
5  ???                            0x0 workLoopSync + 8514 (main.jsbundle:8514:24:8514)
6  ???                            0x0 renderRootSync + 8496 (main.jsbundle:8496:21:8496)
7  ???                            0x0 performSyncWorkOnRoot + 8233 (main.jsbundle:8233:36:8233)
8  ???                            0x0 flushSyncCallbacks + 5165 (main.jsbundle:5165:32:5165)
9  ???                            0x0 scheduleUpdateOnFiber + 8047 (main.jsbundle:8047:345:8047)
10 ???                            0x0 enqueueSetState + 5551 (main.jsbundle:5551:38:5551)
11 ???                            0x0 anonymous + 73232 (main.jsbundle:73232:33:73232)
12 ???                            0x0 anonymous + 63722 (main.jsbundle:63722:47:63722)
13 ???                            0x0 each + 61811 (main.jsbundle:61811:19:61811)
14 ???                            0x0 keyChanged + 63704 (main.jsbundle:63704:29:63704)
15 ???                            0x0 anonymous + 63943 (main.jsbundle:63943:19:63943)
16 ???                            0x0 each + 61805 (main.jsbundle:61805:19:61805)
17 ???                            0x0 anonymous + 63942 (main.jsbundle:63942:31:63942)
18 ???                            0x0 tryCallOne + 53 (InternalBytecode.js:53:16:53)
19 ???                            0x0 anonymous + 139 (InternalBytecode.js:139:27:139)
20 ???                            0x0 apply ((native):0:0)
21 ???                            0x0 anonymous + 11428 (main.jsbundle:11428:26:11428)
22 ???                            0x0 _callTimer + 11336 (main.jsbundle:11336:17:11336)
23 ???                            0x0 _callReactNativeMicrotasksPass + 11367 (main.jsbundle:11367:17:11367)
24 ???                            0x0 callReactNativeMicrotasks + 11573 (main.jsbundle:11573:44:11573)
25 ???                            0x0 __callReactNativeMicrotasks + 3189 (main.jsbundle:3189:46:3189)
26 ???                            0x0 anonymous + 3064 (main.jsbundle:3064:45:3064)
27 ???                            0x0 __guard + 3172 (main.jsbundle:3172:15:3172)
28 ???                            0x0 flushedQueue + 3063 (main.jsbundle:3063:21:3063)
29 ???                            0x0 invokeCallbackAndReturnFlushedQueue + 3056 (main.jsbundle:3056:33:3056)

Device Model:iPhone XS Orientation: Portrait RAM free: 0 B Disk free: 0 B

Operating System Version:15.5.0 Orientation: Portrait Jailbroken:No

Crash Date:Jun 13, 2022, 5:42:51 PM App version:1.1.62 (1.1.62.0)

[kidroca]

Actually the mismapping resulted from the stack trace format

I'm not sure this stack trace format is valid:

main.jsbundle:199284:30:199284

What works is

main.jsbundle:199284:30

e.g.

text:line:col

The way metro-symbolicate works is it looks for these :line:col pairs and replaces content with original source When it sees 199284:30:199284 it treats 30 as line and 199284 as column

---

Another thing that breaks mapping are lines like this one

20 ???                            0x0 apply ((native):0:0)
                                                      ^

Columns can start with 0 but line numbers should start from 1 or higher 0 for line number result in metro-symbolicate exiting with error code almost immediately (We can use --input-line-start 0 but it results in mismatched and unmapped sources so it's not an option)

---

With all that said, here's the result

Input

• ios
• v1.1.62-0

0  ???                            0x0 shouldComponentUpdate + 199284 (main.jsbundle:199284:30)
1  ???                            0x0 checkShouldComponentUpdate + 5581 (main.jsbundle:5581:109)
2  ???                            0x0 updateClassComponent + 6842 (main.jsbundle:6842:382)
3  ???                            0x0 beginWork$1 + 8980 (main.jsbundle:8980:226)
4  ???                            0x0 performUnitOfWork + 8525 (main.jsbundle:8525:27)
5  ???                            0x0 workLoopSync + 8514 (main.jsbundle:8514:24)
6  ???                            0x0 renderRootSync + 8496 (main.jsbundle:8496:21)
7  ???                            0x0 performSyncWorkOnRoot + 8233 (main.jsbundle:8233:36)
8  ???                            0x0 flushSyncCallbacks + 5165 (main.jsbundle:5165:32)
9  ???                            0x0 scheduleUpdateOnFiber + 8047 (main.jsbundle:8047:345)
10 ???                            0x0 enqueueSetState + 5551 (main.jsbundle:5551:38)
11 ???                            0x0 anonymous + 73232 (main.jsbundle:73232:33)
12 ???                            0x0 anonymous + 63722 (main.jsbundle:63722:47)
13 ???                            0x0 each + 61811 (main.jsbundle:61811:19)
14 ???                            0x0 keyChanged + 63704 (main.jsbundle:63704:29)
15 ???                            0x0 anonymous + 63943 (main.jsbundle:63943:19)
16 ???                            0x0 each + 61805 (main.jsbundle:61805:19)
17 ???                            0x0 anonymous + 63942 (main.jsbundle:63942:31)
18 ???                            0x0 tryCallOne + 53 (InternalBytecode.js:53:1)
19 ???                            0x0 anonymous + 139 (InternalBytecode.js:139:27)
21 ???                            0x0 anonymous + 11428 (main.jsbundle:11428:26)
22 ???                            0x0 _callTimer + 11336 (main.jsbundle:11336:17)
23 ???                            0x0 _callReactNativeMicrotasksPass + 11367 (main.jsbundle:11367:17)
24 ???                            0x0 callReactNativeMicrotasks + 11573 (main.jsbundle:11573:44)
25 ???                            0x0 __callReactNativeMicrotasks + 3189 (main.jsbundle:3189:46)
26 ???                            0x0 anonymous + 3064 (main.jsbundle:3064:45)
27 ???                            0x0 __guard + 3172 (main.jsbundle:3172:15)
28 ???                            0x0 flushedQueue + 3063 (main.jsbundle:3063:21)
29 ???                            0x0 invokeCallbackAndReturnFlushedQueue + 3056 (main.jsbundle:3056:33)

Decoded stacks

0  ???                            0x0 shouldComponentUpdate + 199284 (src/pages/home/report/ReportActionsView.js:114:constructor)
1  ???                            0x0 checkShouldComponentUpdate + 5581 (node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:2527:checkShouldComponentUpdate)
2  ???                            0x0 updateClassComponent + 6842 (node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:4461:updateClassComponent)
3  ???                            0x0 beginWork$1 + 8980 (node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:7355:beginWork$1)
4  ???                            0x0 performUnitOfWork + 8525 (node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:6771:performUnitOfWork)
5  ???                            0x0 workLoopSync + 8514 (node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:6764:workLoopSync)
6  ???                            0x0 renderRootSync + 8496 (node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:6746:renderRootSync)
7  ???                            0x0 performSyncWorkOnRoot + 8233 (node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:6454:performSyncWorkOnRoot)
8  ???                            0x0 flushSyncCallbacks + 5165 (node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:2058:flushSyncCallbacks)
9  ???                            0x0 scheduleUpdateOnFiber + 8047 (node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:6211:scheduleUpdateOnFiber)
10 ???                            0x0 enqueueSetState + 5551 (node_modules/react-native/Libraries/Renderer/implementations/ReactNativeRenderer-prod.js:2489:classComponentUpdater.enqueueSetState)
11 ???                            0x0 anonymous + 73232 (node_modules/react-native-onyx/node_modules/react/cjs/react.production.min.js:13:E.prototype.setState)
12 ???                            0x0 anonymous + 63722 (node_modules/react-native-onyx/lib/Onyx.js:339:_.each$argument_1)
13 ???                            0x0 each + 61811 (node_modules/underscore/underscore-umd.js:1335:each)
14 ???                            0x0 keyChanged + 63704 (node_modules/react-native-onyx/lib/Onyx.js:319:keyChanged)
15 ???                            0x0 anonymous + 63943 (node_modules/react-native-onyx/lib/Onyx.js:715:_.each$argument_1)
16 ???                            0x0 each + 61805 (node_modules/underscore/underscore-umd.js:1330:each)
17 ???                            0x0 anonymous + 63942 (node_modules/react-native-onyx/lib/Onyx.js:714:getAllKeys.then$argument_0)
18 ???                            0x0 tryCallOne + 53 (node_modules/@react-native-community/cli-plugin-metro/node_modules/metro-runtime/src/polyfills/require.js:149:metroImportDefault)
19 ???                            0x0 anonymous + 139 (node_modules/@react-native-community/cli-plugin-metro/node_modules/metro-runtime/src/polyfills/require.js:259:registerSegment)
21 ???                            0x0 anonymous + 11428 (node_modules/react-native/Libraries/Core/Timers/JSTimers.js:248:_allocateCallback$argument_0)
22 ???                            0x0 _callTimer + 11336 (node_modules/react-native/Libraries/Core/Timers/JSTimers.js:112:_callTimer)
23 ???                            0x0 _callReactNativeMicrotasksPass + 11367 (node_modules/react-native/Libraries/Core/Timers/JSTimers.js:166:_callReactNativeMicrotasksPass)
24 ???                            0x0 callReactNativeMicrotasks + 11573 (node_modules/react-native/Libraries/Core/Timers/JSTimers.js:418:callReactNativeMicrotasks)
25 ???                            0x0 __callReactNativeMicrotasks + 3189 (node_modules/react-native/Libraries/BatchedBridge/MessageQueue.js:393:__callReactNativeMicrotasks)
26 ???                            0x0 anonymous + 3064 (node_modules/react-native/Libraries/BatchedBridge/MessageQueue.js:135:__guard$argument_0)
27 ???                            0x0 __guard + 3172 (node_modules/react-native/Libraries/BatchedBridge/MessageQueue.js:370:__guard)
28 ???                            0x0 flushedQueue + 3063 (node_modules/react-native/Libraries/BatchedBridge/MessageQueue.js:134:__guard$argument_0)
29 ???                            0x0 invokeCallbackAndReturnFlushedQueue + 3056 (node_modules/react-native/Libraries/BatchedBridge/MessageQueue.js:130:invokeCallbackAndReturnFlushedQueue)

[kidroca]

I can submit a PR for the following functionality

package.json commands or scripts for:

• sourcemap:ios - create .map file for iOS (main.jsbundle)
• sourcemap:android - create .map file for Android (index.android.bundle)
  • alternatively, if we only need it after build (for artifacts) the .map can be copied from android/
• symbolicate - use it on a .map file and a piece of stack trace to output decoded stack trace. npm run symbolicate main.jsbundle.map crashlytics.stack

A higher level script can compose these to decode firebase crashlytics data

• use platform and app version to obtain or create a .map file
• pass crashlytics stack traces through symbolicate

[AndrewGable]

We have total control of the stacktraces, so feel free to slice and dice it into what will work for iOS, then I can adjust our internal code to use the tweaked stacktrace.

Any code that needs to run locally when we receive the trace, you can just add to this PR and I will add it to our internal code.

[kidroca]

@AndrewGable I've opened a PR with a script for generating source maps and decoding stack traces: https://github.com/Expensify/App/pull/9596

See if it works for you (more details in the PR)

[AndrewGable]

Ok so - I do like that change, but for reference, we will not be decoding the stacktraces via a GH Action, it will be done via an internal service running PHP. Don't think it changes the solution, but just an FYI.

[roryabraham]

Stumbled upon some documentation on native-level debug symbols for Android: https://developer.android.com/studio/build/shrink-code#native-crash-support

[kidroca]

@AndrewGable I'm not sure if you've checked this out but I've opened an alternative PR which I think might be better

• https://github.com/Expensify/App/pull/9641

The idea is described in the "Details" section but it's basically this

• we generate sourcemaps during build
• we save the generated source maps somewhere where they can be mapped by platform and version

This way we don't need to generate the source-maps from scratch each time Crashlytics capture an error

The same thing can be achieved with the Original PR (capture sourcemaps at build),but the alternative PR is simpler with less changes

[AndrewGable]

I agree that this new proposed solution is much better 👍

[kidroca]

Alright - I've added comprehensive tests steps to let anyone manually generate and test sourcemaps Perhaps we should write some documentation for future reference somewhere?

Source maps are captured during build time (platformDeploy workflow) Then they are stored as artifacts

I think artifacts are suitable and can be fetched by version If you'd like that and decide to stick with them, perhaps we should increase their "life time", otherwise they're deleted after 2 weeks Alternatively the job can just push the source maps somewhere and not capture artifacts

[roryabraham]

Alternatively the job can just push the source maps somewhere and not capture artifacts

The max is 90 days, which might be enough for staging and production. I'd say let's bump the artifact retention period up to its max and not worry about it beyond that. If we wanted we could push the source maps for production builds as artifacts with the release, which AFAIK are kept permanently. It might be a good idea to upload the desktop app .dmg as a release artifact too in case we ever have a bad fire and need an easy way for people to downgrade.

[kidroca]

If you'd like that and decide to stick with them, perhaps we should increase their "life time", otherwise they're deleted after 2 weeks

@roryabraham I was wrong on this one, the default retention period for artifacts is 90 days (same as max): https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration#artifact-and-log-retention-policy

By default, the artifacts and log files generated by workflows are retained for 90 days before they are automatically deleted

I haven't set retention-days for source maps so they'll just use max

[roryabraham]

PR to upload sourcemaps as workflow artifacts merged 👍

There were a few gotchas we'll want to watch out for when we hook this into the PHP <-> Firebase integration, listed here.

@AndrewGable if I'm not mistaken the piece @kidroca can do is done now

[roryabraham]

No update here – will try to prioritize by EOW, but otherwise will do in 2-3 weeks at the earliest if anyone else wants to grab this.

[michaelhaxhiu]

Just spoke with Rory about this IRL, and wanted to chime in to say I think it would be a helpful addition to add this piece of context to the crash GHs because they aren't really getting picked up by contributors.