sentry-cocoa icon indicating copy to clipboard operation
sentry-cocoa copied to clipboard

refactor: Serialize envelope straight to a file

Open itaybre opened this issue 6 months ago • 5 comments

:scroll: Description

Write files straight to a file on storeEnvelope instead of storing the data which may cause increased memory usage

:bulb: Motivation and Context

:green_heart: How did you test it?

:pencil: Checklist

You have to check all boxes before merging:

  • [ ] I added tests to verify the changes.
  • [ ] No new PII added or SDK only sends newly added PII if sendDefaultPII is enabled.
  • [ ] I updated the docs if needed.
  • [ ] I updated the wizard if needed.
  • [ ] Review from the native team if needed.
  • [ ] No breaking change or entry added to the changelog.
  • [ ] No breaking change for hybrid SDKs or communicated to hybrid SDKs.

itaybre avatar Jun 04 '25 23:06 itaybre

Codecov Report

Attention: Patch coverage is 84.42623% with 19 lines in your changes missing coverage. Please review.

Project coverage is 86.122%. Comparing base (5652830) to head (be55d4f).

Files with missing lines Patch % Lines
Sources/Sentry/SentrySerialization.m 84.166% 19 Missing :warning:
Additional details and impacted files

Impacted file tree graph

@@              Coverage Diff              @@
##              main     #5360       +/-   ##
=============================================
- Coverage   86.179%   86.122%   -0.057%     
=============================================
  Files          399       399               
  Lines        34616     34712       +96     
  Branches     14972     15014       +42     
=============================================
+ Hits         29832     29895       +63     
- Misses        4739      4775       +36     
+ Partials        45        42        -3     
Files with missing lines Coverage Δ
Sources/Sentry/SentryFileManager.m 94.420% <100.000%> (-0.324%) :arrow_down:
Sources/Sentry/SentrySerialization.m 93.823% <84.166%> (-5.340%) :arrow_down:

... and 9 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 5652830...be55d4f. Read the comment docs.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov[bot] avatar Jun 04 '25 23:06 codecov[bot]

Performance metrics :rocket:

  Plain With Sentry Diff
Startup time 1214.78 ms 1229.67 ms 14.89 ms
Size 23.75 KiB 849.02 KiB 825.27 KiB

Baseline results on branch: main

Startup times

Revision Plain With Sentry Diff
077e94086236b0be69cb0982b64053b94f2c9ea4 1227.31 ms 1246.51 ms 19.20 ms
15b8c6111854bacf68624e765d99adcf93bfc987 1223.16 ms 1244.83 ms 21.67 ms
bc0c36e91c733f434511cb832b636fdc8ad90330 1220.41 ms 1239.45 ms 19.04 ms
455619dd977fcfe15f8d8294a8d217fcb0b90d15 1231.40 ms 1237.70 ms 6.30 ms
f8833c443d047f0c8dac5710e52b4c0624fc52ce 1236.45 ms 1252.82 ms 16.37 ms
728804f93e9f74d591108dd8a8d2df61a645c02b 1235.26 ms 1254.20 ms 18.94 ms
6e342acf0107a6421fa9221f9ecd93c5f3ef0b3b 1220.24 ms 1240.14 ms 19.90 ms
46f5eb8f90236e0b538f5ff1c237e290c12a25e6 1200.09 ms 1231.38 ms 31.29 ms
3912b16e2a203c2dd8155aa48aaf4c4dc10199e3 1230.54 ms 1249.65 ms 19.11 ms
061c982c8616fa6ca7f0762998f39a8f4d8d2fc2 1226.33 ms 1243.65 ms 17.33 ms

App size

Revision Plain With Sentry Diff
077e94086236b0be69cb0982b64053b94f2c9ea4 21.58 KiB 706.97 KiB 685.39 KiB
15b8c6111854bacf68624e765d99adcf93bfc987 20.76 KiB 419.67 KiB 398.91 KiB
bc0c36e91c733f434511cb832b636fdc8ad90330 22.31 KiB 775.27 KiB 752.95 KiB
455619dd977fcfe15f8d8294a8d217fcb0b90d15 20.76 KiB 432.87 KiB 412.11 KiB
f8833c443d047f0c8dac5710e52b4c0624fc52ce 21.58 KiB 422.66 KiB 401.08 KiB
728804f93e9f74d591108dd8a8d2df61a645c02b 22.85 KiB 411.76 KiB 388.91 KiB
6e342acf0107a6421fa9221f9ecd93c5f3ef0b3b 20.76 KiB 436.66 KiB 415.90 KiB
46f5eb8f90236e0b538f5ff1c237e290c12a25e6 20.76 KiB 432.37 KiB 411.61 KiB
3912b16e2a203c2dd8155aa48aaf4c4dc10199e3 21.58 KiB 625.82 KiB 604.24 KiB
061c982c8616fa6ca7f0762998f39a8f4d8d2fc2 21.58 KiB 683.51 KiB 661.93 KiB

Previous results on branch: itay/cocoa-243-fix-nsmallocexception-when-writing-envelope-to-disk

Startup times

Revision Plain With Sentry Diff
d3acd4be4f76c377e5fa4d42c5536ffa84130be9 1223.69 ms 1239.63 ms 15.94 ms
bf1998e1325b612d410fd79186b0762696470a24 1227.84 ms 1250.59 ms 22.76 ms
866822342e7c765c0b1289b5563be141d45fedf3 1212.33 ms 1232.67 ms 20.33 ms
9ac31f2b05a31afd8f74bc821670c62fe24648a6 1228.12 ms 1251.50 ms 23.38 ms

App size

Revision Plain With Sentry Diff
d3acd4be4f76c377e5fa4d42c5536ffa84130be9 23.76 KiB 838.38 KiB 814.62 KiB
bf1998e1325b612d410fd79186b0762696470a24 23.75 KiB 848.95 KiB 825.20 KiB
866822342e7c765c0b1289b5563be141d45fedf3 23.76 KiB 838.63 KiB 814.87 KiB
9ac31f2b05a31afd8f74bc821670c62fe24648a6 23.76 KiB 838.99 KiB 815.23 KiB

github-actions[bot] avatar Jun 04 '25 23:06 github-actions[bot]

I don't have the bandwidth to review this PR today, and I'm on PTO the whole next week. FYI, I tried something similar with a NSFileHandle last year and we had a very nasty severe incident https://github.com/getsentry/sentry-cocoa/pull/4219 with corrupted envelopes. Check the changelog of 8.35.0 and the 8.35.1. Serializing envelopes is a core part of the SDK. When we do this we have to be 100% confident that everything is working correctly. With CI being so flaky right now, I highly doubt that we can be so confident.

While it's highly admirable that you're trying to improve this, I would rather focus on making the CI stable and postpone such refactorings for now, please.

philipphofmann avatar Jun 06 '25 09:06 philipphofmann

I don't have the bandwidth to review this PR today, and I'm on PTO the whole next week. FYI, I tried something similar with a NSFileHandle last year and we had a very nasty severe incident #4219 with corrupted envelopes. Check the changelog of 8.35.0 and the 8.35.1. Serializing envelopes is a core part of the SDK. When we do this we have to be 100% confident that everything is working correctly. With CI being so flaky right now, I highly doubt that we can be so confident.

While it's highly admirable that you're trying to improve this, I would rather focus on making the CI stable and postpone such refactorings for now, please.

I agree it is a risky change, and to be honest, I would like to refactor how we read envelopes too. It think the code can be improved and add some failsafes there.

Regarding the CI flakiness, we are improving there 💪

itaybre avatar Jun 06 '25 16:06 itaybre

🚨 Detected changes in high risk code 🚨

High-risk code can easily blow up and is hard to test. We had severe bugs in the past. Be extra careful when changing these files, and have an extra careful look at these:

  • [ ] Sources/Sentry/SentryFileManager.m
  • [ ] Sources/Sentry/SentrySerialization.m

github-actions[bot] avatar Jun 20 '25 15:06 github-actions[bot]

Should this PR be marked as draft again?

philprime avatar Jul 04 '25 11:07 philprime