refactor: Serialize envelope straight to a file
:scroll: Description
Write files straight to a file on storeEnvelope instead of storing the data which may cause increased memory usage
:bulb: Motivation and Context
:green_heart: How did you test it?
:pencil: Checklist
You have to check all boxes before merging:
- [ ] I added tests to verify the changes.
- [ ] No new PII added or SDK only sends newly added PII if
sendDefaultPIIis enabled. - [ ] I updated the docs if needed.
- [ ] I updated the wizard if needed.
- [ ] Review from the native team if needed.
- [ ] No breaking change or entry added to the changelog.
- [ ] No breaking change for hybrid SDKs or communicated to hybrid SDKs.
Codecov Report
Attention: Patch coverage is 84.42623% with 19 lines in your changes missing coverage. Please review.
Project coverage is 86.122%. Comparing base (
5652830) to head (be55d4f).
| Files with missing lines | Patch % | Lines |
|---|---|---|
| Sources/Sentry/SentrySerialization.m | 84.166% | 19 Missing :warning: |
Additional details and impacted files
@@ Coverage Diff @@
## main #5360 +/- ##
=============================================
- Coverage 86.179% 86.122% -0.057%
=============================================
Files 399 399
Lines 34616 34712 +96
Branches 14972 15014 +42
=============================================
+ Hits 29832 29895 +63
- Misses 4739 4775 +36
+ Partials 45 42 -3
| Files with missing lines | Coverage Δ | |
|---|---|---|
| Sources/Sentry/SentryFileManager.m | 94.420% <100.000%> (-0.324%) |
:arrow_down: |
| Sources/Sentry/SentrySerialization.m | 93.823% <84.166%> (-5.340%) |
:arrow_down: |
... and 9 files with indirect coverage changes
Continue to review full report in Codecov by Sentry.
Legend - Click here to learn more
Δ = absolute <relative> (impact),ø = not affected,? = missing dataPowered by Codecov. Last update 5652830...be55d4f. Read the comment docs.
:rocket: New features to boost your workflow:
- :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
Performance metrics :rocket:
| Plain | With Sentry | Diff | |
|---|---|---|---|
| Startup time | 1214.78 ms | 1229.67 ms | 14.89 ms |
| Size | 23.75 KiB | 849.02 KiB | 825.27 KiB |
Baseline results on branch: main
Startup times
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 077e94086236b0be69cb0982b64053b94f2c9ea4 | 1227.31 ms | 1246.51 ms | 19.20 ms |
| 15b8c6111854bacf68624e765d99adcf93bfc987 | 1223.16 ms | 1244.83 ms | 21.67 ms |
| bc0c36e91c733f434511cb832b636fdc8ad90330 | 1220.41 ms | 1239.45 ms | 19.04 ms |
| 455619dd977fcfe15f8d8294a8d217fcb0b90d15 | 1231.40 ms | 1237.70 ms | 6.30 ms |
| f8833c443d047f0c8dac5710e52b4c0624fc52ce | 1236.45 ms | 1252.82 ms | 16.37 ms |
| 728804f93e9f74d591108dd8a8d2df61a645c02b | 1235.26 ms | 1254.20 ms | 18.94 ms |
| 6e342acf0107a6421fa9221f9ecd93c5f3ef0b3b | 1220.24 ms | 1240.14 ms | 19.90 ms |
| 46f5eb8f90236e0b538f5ff1c237e290c12a25e6 | 1200.09 ms | 1231.38 ms | 31.29 ms |
| 3912b16e2a203c2dd8155aa48aaf4c4dc10199e3 | 1230.54 ms | 1249.65 ms | 19.11 ms |
| 061c982c8616fa6ca7f0762998f39a8f4d8d2fc2 | 1226.33 ms | 1243.65 ms | 17.33 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 077e94086236b0be69cb0982b64053b94f2c9ea4 | 21.58 KiB | 706.97 KiB | 685.39 KiB |
| 15b8c6111854bacf68624e765d99adcf93bfc987 | 20.76 KiB | 419.67 KiB | 398.91 KiB |
| bc0c36e91c733f434511cb832b636fdc8ad90330 | 22.31 KiB | 775.27 KiB | 752.95 KiB |
| 455619dd977fcfe15f8d8294a8d217fcb0b90d15 | 20.76 KiB | 432.87 KiB | 412.11 KiB |
| f8833c443d047f0c8dac5710e52b4c0624fc52ce | 21.58 KiB | 422.66 KiB | 401.08 KiB |
| 728804f93e9f74d591108dd8a8d2df61a645c02b | 22.85 KiB | 411.76 KiB | 388.91 KiB |
| 6e342acf0107a6421fa9221f9ecd93c5f3ef0b3b | 20.76 KiB | 436.66 KiB | 415.90 KiB |
| 46f5eb8f90236e0b538f5ff1c237e290c12a25e6 | 20.76 KiB | 432.37 KiB | 411.61 KiB |
| 3912b16e2a203c2dd8155aa48aaf4c4dc10199e3 | 21.58 KiB | 625.82 KiB | 604.24 KiB |
| 061c982c8616fa6ca7f0762998f39a8f4d8d2fc2 | 21.58 KiB | 683.51 KiB | 661.93 KiB |
Previous results on branch: itay/cocoa-243-fix-nsmallocexception-when-writing-envelope-to-disk
Startup times
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| d3acd4be4f76c377e5fa4d42c5536ffa84130be9 | 1223.69 ms | 1239.63 ms | 15.94 ms |
| bf1998e1325b612d410fd79186b0762696470a24 | 1227.84 ms | 1250.59 ms | 22.76 ms |
| 866822342e7c765c0b1289b5563be141d45fedf3 | 1212.33 ms | 1232.67 ms | 20.33 ms |
| 9ac31f2b05a31afd8f74bc821670c62fe24648a6 | 1228.12 ms | 1251.50 ms | 23.38 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| d3acd4be4f76c377e5fa4d42c5536ffa84130be9 | 23.76 KiB | 838.38 KiB | 814.62 KiB |
| bf1998e1325b612d410fd79186b0762696470a24 | 23.75 KiB | 848.95 KiB | 825.20 KiB |
| 866822342e7c765c0b1289b5563be141d45fedf3 | 23.76 KiB | 838.63 KiB | 814.87 KiB |
| 9ac31f2b05a31afd8f74bc821670c62fe24648a6 | 23.76 KiB | 838.99 KiB | 815.23 KiB |
I don't have the bandwidth to review this PR today, and I'm on PTO the whole next week. FYI, I tried something similar with a NSFileHandle last year and we had a very nasty severe incident https://github.com/getsentry/sentry-cocoa/pull/4219 with corrupted envelopes. Check the changelog of 8.35.0 and the 8.35.1. Serializing envelopes is a core part of the SDK. When we do this we have to be 100% confident that everything is working correctly. With CI being so flaky right now, I highly doubt that we can be so confident.
While it's highly admirable that you're trying to improve this, I would rather focus on making the CI stable and postpone such refactorings for now, please.
I don't have the bandwidth to review this PR today, and I'm on PTO the whole next week. FYI, I tried something similar with a NSFileHandle last year and we had a very nasty severe incident #4219 with corrupted envelopes. Check the changelog of 8.35.0 and the 8.35.1. Serializing envelopes is a core part of the SDK. When we do this we have to be 100% confident that everything is working correctly. With CI being so flaky right now, I highly doubt that we can be so confident.
While it's highly admirable that you're trying to improve this, I would rather focus on making the CI stable and postpone such refactorings for now, please.
I agree it is a risky change, and to be honest, I would like to refactor how we read envelopes too. It think the code can be improved and add some failsafes there.
Regarding the CI flakiness, we are improving there 💪
🚨 Detected changes in high risk code 🚨
High-risk code can easily blow up and is hard to test. We had severe bugs in the past. Be extra careful when changing these files, and have an extra careful look at these:
- [ ] Sources/Sentry/SentryFileManager.m
- [ ] Sources/Sentry/SentrySerialization.m
Should this PR be marked as draft again?