Performance issue with micrometer and otel baggage

[SinhaAmit]

Describe the bug I'm attempting to add several OpenTelemetry baggage items to the current tracing context using Micrometer and OpenTelemetry in a spring-cloud-starter-gateway-server-webflux application, but this appears to be significantly impacting throughput. It degrades from ~108 RPS to ~51 RPS 😟

Before baggage:

After baggage:

I have followed the steps given in the micrometer documentation page, i.e. by using ReactorBaggage and ObservationAwareBaggageThreadLocalAccessor

see here - https://docs.micrometer.io/tracing/reference/configuring.html#_context_propagation_with_micrometer_tracing

Sample

id("org.springframework.boot") version "3.5.0"
id("io.spring.dependency-management") version "1.1.7"

extra["springCloudVersion"] = "2025.0.0"


implementation("io.micrometer:micrometer-tracing")
implementation("io.micrometer:micrometer-tracing-bridge-otel")

Registering ObservationAwareBaggageThreadLocalAccessor:

@Bean
    fun otelWebFilter(
        tracer: Tracer,
        registry: ObservationRegistry,
    ): OtelWebFilter {
        ContextRegistry.getInstance()
            .registerThreadLocalAccessor(ObservationAwareBaggageThreadLocalAccessor(registry, tracer))
        ContextRegistry.getInstance().registerThreadLocalAccessor(ObservationAwareSpanThreadLocalAccessor(tracer))
        return OtelWebFilter(tracer)
    }

ReactorBaggage with contextWrite:

class OtelWebFilter(private val tracer: Tracer) : WebFilter {

    private val logger = LoggerFactory.getLogger(OtelWebFilter::class.java)

    override fun filter(
        exchange: ServerWebExchange,
        chain: WebFilterChain
    ): Mono<Void> {
        val baggage = mapOf(
            "username" to exchange.request.headers.getFirst("X-Username").orEmpty(),
            "tenantId" to exchange.request.headers.getFirst("X-Tenant-Id").orEmpty(),
        )
        return chain.filter(exchange)
            .doOnSubscribe { logger.debug("Setting baggage for path: {}", exchange.request.path) }
            .doOnSuccess { logger.debug("Setting baggage completed") }
            .contextWrite(ReactorBaggage.append(baggage))
    }
}

Please look into it and let me know if you need further details

[spencergibb]

@jonatan-ivanov

[jonatan-ivanov]

Shouldn't this work just by setting a few properties? See Boot baggage docs and Boot baggage properties.

Do you see performance degradation if you do this?

[SinhaAmit]

Hi @jonatan-ivanov, I need to add a few custom baggage items for all gateway calls. Instead of adding a header value (as shown in the code snippet above in the issue, which is just for illustration), there is no issue when baggage properties are used.

However, when I create a baggage by following Boot baggage docs, I have the same traces again and again (even though scope is closed):

Note: I see a debug log as Trying to close scope which does not represent current context. Ignoring the call. on scope close, which is a bit strange.

Below is the updated code now:

@Component
class CreatingBaggage(private val tracer: Tracer) {


    fun create(key: String, value: String): BaggageInScope {
        return tracer.createBaggageInScope(key, value)
    }
}

class OtelWebFilter(private val creatingBaggage: CreatingBaggage) : WebFilter {


    override fun filter(
        exchange: ServerWebExchange,
        chain: WebFilterChain
    ): Mono<Void> {

        val scopes = mutableListOf<BaggageInScope>()
        return chain.filter(exchange)
           .doOnSubscribe {
                scopes += creatingBaggage.create("username", exchange.request.headers.getFirst("X-Username").orEmpty())
                scopes += creatingBaggage.create("tenantId", exchange.request.headers.getFirst("X-Tenant-Id").orEmpty())
            }
            .doFinally { scopes.forEach { it.close() } }
    }
}

[jonatan-ivanov]

I need to add a few custom baggage items for all gateway calls. Instead of adding a header value (as shown in the code snippet above in the issue, which is just for illustration)

I think it would help showing in your examples what you really want to do instead of what you don't. :)

However, when I create a baggage by following Boot baggage docs, I have the same traces again and again (even though scope is closed):

So are you doing this (from the Boot docs)? Because your example above is very different.

try (BaggageInScope scope = this.tracer.createBaggageInScope("baggage1", "value1")) {
    // code
}

With Reactor, you should use .contextWrite instead, see docs: https://docs.micrometer.io/tracing/reference/configuring.html#_context_propagation_with_micrometer_tracing

(With Boot you don't need to call .enableAutomaticContextPropagation and I guess neither . registerThreadLocalAccessor, Boot should auto-configure those for you.)

In your second example you are still getting the baggage from the headers and in the same comment you just said this is not what you want to do. Could you please provide us a minimal sample Java project to reproduce this issue so we can more easily investigate and ensure any fix is working properly for your use case? Also, to ensure we don't have the https://xyproblem.info/

[SinhaAmit]

Hi @jonatan-ivanov, here is the reproducer for this issue - https://github.com/SinhaAmit/gateway-reproducer (follow the readme to setup)

• For now, I have commented out the usage of ReactorBaggage - causes performance issue (possibly due to use of ConcurrentHashMap inside ObservationAwareBaggageThreadLocalAccessor)
• I followed the Boot baggage docs and the created baggage fields are correctly included in logs as MDC entries in non-reactive code. However, they don't appear in the logs when using reactive code.

[SinhaAmit]

@jonatan-ivanov any update on this?