Significant memory usage

[RobicToNieMaKomu]

Integration

sentry-spring-boot-starter-jakarta

Java Version

17

Version

7.1.0

Steps to Reproduce

Our service is running on SB 3.1.8 & reactive stack (Webflux).

1. Baseline. Without Sentry, memory consumption oscillates around 700 Mb (committed ~900 Mb, and max is 1.88 Gb).
2. After enabling integration on the defaults (with tracing disabled), mem usage went up to the ceiling

sentry:
  dsn: ...
  enabled: false
  enableTracing: false
  environment: dev
  serverName: server
  shutdownTimeout: 0

3. After further tuning, I was able to reduce the memory:

      SENTRY_MAXBREADCRUMBS: 1
      SENTRY_ENABLEUSERINTERACTIONBREADCRUMBS: false
      SENTRY_DIAGNOSTICLEVEL: fatal
      SENTRY_MAXCACHEITEMS: 0
      SENTRY_ATTACHSTACKTRACE: false

However, the situation is still unacceptable, as integration increased RAM usage by 50% (!). Can we do something to make it better? Can breadcrumbs be entirely disabled? They are useless in our case as they are not correlated with the request causing an actual error.

Expected Result

Sane memory/CPU footprint.

Actual Result

Gigantic memory consumption.

[adinauer]

@RobicToNieMaKomu thanks for opening this issue. We'll take a look and update here.

[adinauer]

@RobicToNieMaKomu I can confirm, we're using up more memory on startup but after manually triggering GC, it ends up roughly the same.

How I tested

• I fired up our Spring Boot 3 Webflux sample
• let it startup for a bit then manually triggered GC in visualvm a couple times
• sent some requests in using wrk -t2 -c4 -d30s -H "Authorization: Basic dXNlcjpwYXNzd29yZA==" http://127.0.0.1:8080/todo-webclient/1. This sends ~ 4k requests in 30s:

Running 30s test @ http://127.0.0.1:8080/todo-webclient/1
  2 threads and 4 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    28.21ms    9.48ms 174.51ms   93.47%
    Req/Sec    72.24     11.44   101.00     68.91%
  4325 requests in 30.03s, 532.18KB read
Requests/sec:    144.01
Transfer/sec:     17.72KB

• manually triggered GC a couple times again
• reran wrk
• manually triggered GC a couple times again

The endpoint does not record any errors but did create transactions and spans (when Sentry was enabled).

Results

Here's memory with Sentry:

And memory without Sentry:

So with Sentry enabled the server is consuming a lot more memory on startup but it is able to free that back up. Will keep investigating.

Questions

Can you please tell us a bit about your usage of Sentry on the screenshots you provided?

• Do you do any manual capturing of errors, breadcrumbs etc.?
• Other interaction with the Sentry API?
• Are any/many exceptions thrown and recorded by Sentry?
• Roughly how many requests is the server seeing?
• Do you use CRONS? There's another potential memory problem there, see https://github.com/getsentry/sentry-java/issues/3165

[adinauer]

@RobicToNieMaKomu if you're not using any of our AOP features (@SentryTransaction, @SentrySpan, @SentryCheckIn annotations) you could try to disable those using sentry.enable-aot-compatibility=true and see if that helps lower the initial memory consumption. From my testing it only used ~ 1/3 of memory on startup this way.

[adinauer]

Here's another screenshot showing memory consumption with above mentioned test procedure and sentry.enable-aot-compatibility=true:

[RobicToNieMaKomu]

Hey @adinauer , thank's for looking into that!

Do you do any manual capturing of errors, breadcrumbs etc.?

We don't create/capture/use breadcrumbs. We catch exceptions above the controller's layer in the @ControllerAdvice class -> @ExceptionHandler methods.

Other interaction with the Sentry API?

Nope.

Are any/many exceptions thrown and recorded by Sentry?

Not many, during a few days, we got <100 occurrences of 3 different Exceptions

Roughly how many requests is the server seeing?

Single pod/instance handles roughly up to 150 RPS

Do you use CRONS?

Nope, at least not directly in our code (not sure if it's used inside Spring itself)

I just set sentry.enable-aot-compatibility=true, will report back tomorrow with the result.

[adinauer]

@RobicToNieMaKomu you can also try playing with these settings:

sentry.logging.minimum-event-level=info
sentry.logging.minimum-breadcrumb-level=debug

and you could drop all breadcrumbs from BeforeBreadcrumbCallback. Should be as simple as providing a spring bean that always returns null from Breadcrumb execute(@NotNull Breadcrumb breadcrumb, @NotNull Hint hint).

[RobicToNieMaKomu]

This is what it looks like after applying these properties:

sentry.enable-aot-compatibility=true
sentry.logging.minimum-event-level=info
sentry.logging.minimum-breadcrumb-level=debug

and BeforeBreadcrumbCallback -> null:

Maybe memory consumption became slightly smaller, but it still has a significant impact.

[adinauer]

@RobicToNieMaKomu can you please provide a sample that allows us to reprodue the issue, maybe by forking the repo and adding to our Spring Boot 3 Webflux Sample. I'm assuming you're using additional Spring features, settings or other libraries that could cause this when combined with Sentry.

and BeforeBreadcrumbCallback -> null

I'm not sure we're on the same page here. Did you provide a callback that returns null (this is what is needed to drop all breadcrumbs) or did you set the callback to null (this is the default behaviour of not dropping breadcrumbs)?

sentry.logging.minimum-breadcrumb-level=debug

I meant to suggest you can increase the level to not have as many breadcrumbs but it's not that relevant if you're dropping all breadcrumbs anyways or limit breadcrumb count to 1.

[adinauer]

Oh and are you using any special garbage collector settings?

[markushi]

Hey @RobicToNieMaKomu, did you had some time to look into the questions from above^, by any chance?

[RobicToNieMaKomu]

@adinauer I'm sorry, I can't fork the repo (NDA) and don't have enough time to come up with PoC now. The simplified stack we're using in that service:

        <java.version>17</java.version>
        <lombok.version>1.18.30</lombok.version>
        <springboot-cloud.version>2022.0.5</springboot-cloud.version>
        <resilience4j.version>2.1.0</resilience4j.version>
        <protobuf-java.version>3.22.2</protobuf-java.version>
        <awaitility.version>4.2.0</awaitility.version>

Besides, it heavily uses Redis (spring-boot-starter-data-redis-reactive dependency of SB 3.1.8).

I'm not sure we're on the same page here. Did you provide a callback that returns null (this is what is needed to drop all breadcrumbs) or did you set the callback to null (this is the default behaviour of not dropping breadcrumbs)?

I provided a callback that returns null.

are you using any special garbage collector settings?

Java options: -Xss512k -XX:+UseZGC -XX:MaxDirectMemorySize=64m -XX:ReservedCodeCacheSize=120M

[adinauer]

Thanks for the update.