sentry-java icon indicating copy to clipboard operation
sentry-java copied to clipboard

Spring Boot (Webflux): many unrelated http breadcrumbs are recorded with events

Open s-volkov-1 opened this issue 3 years ago • 5 comments

Platform:

  • [x] Spring Boot (Webflux server & WebClient) -> 2.5.5

IDE:

  • [x] IntelliJ -> 2021.2

Build system:

  • [x] Maven -> 3.6.3-jdk-11

Platform installed with:

  • [x] Maven Central

The version of the SDK: 5.2.4


I have the following issue:

Many (tens or even ~hundred) unrelated http breadcrumbs recorded with events. These breadcrumbs are:

  • Incoming server http requests (to actuator endpoints, for example)
  • Outgoing http client requests to external services, but they are old and/or unrelated to original incoming server request

Steps to reproduce:

  • Use spring-boot webflux server application
  • Use sentry-spring-boot-starter
  • Use actual sentry server (probably won't work with NoOpHub)
  • Start app locally
  • Put some load on application
    • I used 50 threads in jmeter accessing actuator health, info and prometheus endpoints without delays
    • prometheus worked better
  • Make some meaningful requests which should cause some kind of error which should be sent to sentry
  • Observe events on sentry server or via debugging SentryClient (less effective)

Actual result: Screenshot from actual server-running instance sentry webflux bug

Expected result: Only a couple of related http breadcrumbs are present

Personal thoughts Both SentryWebFilter & SentrySpanClientWebRequestFilter use HubAdapter, which uses static Sentry.getCurrentHub() method, which returns ThreadLocal instance of Hub.

It seems like this mechanism may not work correctly with reactive application, which serves requests on a small number of threads, and they are switched frequently.

As far as I know, there is a way of storing arbitrary request-related data in webflux:

  • org.springframework.web.server.ServerWebExchange.getAttributes()
  • org.springframework.web.reactive.function.client.ClientRequest.attributes()

s-volkov-1 avatar Nov 02 '21 10:11 s-volkov-1

cc @maciejwalkowiak

marandaneto avatar Nov 02 '21 10:11 marandaneto

Based on ThreadLocal, this issue seems unsolvable.

hu-chia avatar Mar 05 '22 10:03 hu-chia

We have just released 6.15.0 of the Java SDK. If you're on Spring Boot 3.0.3+ you can give ThreadLocalAccessor a try.

Here's a copy of the changelog so you know what's required: Implement ThreadLocalAccessor for propagating Sentry hub with reactor / WebFlux (https://github.com/getsentry/sentry-java/pull/2570)

  • Requires io.micrometer:context-propagation:1.0.2+ as well as Spring Boot 3.0.3+
  • Enable the feature by setting sentry.reactive.thread-local-accessor-enabled=true
  • This is still considered experimental. Once we have enough feedback we may turn this on by default.
  • Checkout the sample here: https://github.com/getsentry/sentry-java/tree/main/sentry-samples/sentry-samples-spring-boot-webflux-jakarta
  • A new hub is now cloned from the main hub for every request

We've also fixed a bug for the WebFlux integration (#2580) in this release for older versions of Spring Boot and when not using context-propagation. In case you give it a try it would be great if you could give us feedback 🙏 .

adinauer avatar Mar 02 '23 13:03 adinauer

Closing this now as it should be resolved (see comment above), feel free to reopen if you run into the problem again.

adinauer avatar May 09 '23 04:05 adinauer

This is still an issue for our Spring Boot 2 integration.

We could take another stab at it by trying to use reactor.core.CoreSubscriber and Hooks.onEachOperator as described here. Needs testing whether this is has a lot of impact on performance.

Putting this in the backlog but can't say when we'll get to it.

adinauer avatar Apr 11 '24 04:04 adinauer