sentry-cli icon indicating copy to clipboard operation
sentry-cli copied to clipboard

Reduce memory usage of sourcemap uploads

Open szokeasaurusrex opened this issue 11 months ago • 0 comments

In addition to being affected by the memory-inefficiency of chunk uploading (#2291), sourcemap uploads are affected by even more severe memory usage issues.

From investigating a sourcemap upload with a memory profiler, I observed three large memory allocations which we should attempt to fix. Note that these tests were performed with --no-rewrite (more info on this later):

  • [ ] The first (and longest) allocation reads all of the sourcemaps into memory. This allocation lasts for pretty much the entire program run. We perform the allocation on this line, which we call when collecting sourcemaps for upload.
    • We can probably address this allocation by memory-mapping the sourcemap files.
    • However, we perform some processing/validation with the sourcemap crate. The crate's documentation states that it loads the entire sourcemap into memory, so we might end up still needing at least to allocate one sourcemap at a time into memory, even after addressing this problem.
  • [X] The second allocation occurs when we clone all of the in-memory sourcemaps in preparation for upload. See #2342. Fixed in #2343.
  • [x] The final allocation occurs in symbolic when we validate that the sourcemap is valid UTF-8. Fix is here: https://github.com/getsentry/symbolic/pull/890 and https://github.com/getsentry/sentry-cli/pull/2346

Lastly, without --no-rewrite, memory usage seems to be even worse. We still need to investigate the case where --no-rewrite is not passed to Sentry CLI.

szokeasaurusrex avatar Jan 15 '25 11:01 szokeasaurusrex