Enabling tracing support with Jaeger provides an repeating error "[ERROR] trace: Failed to update baggage restrictions...""

[renta]

Branch/Environment/Version

• Branch/Version: v3.2.2
• Environment: OSS in the k8s cluster

Describe the bug After enabling a tracing support in an official container tyk-gateway:v3.2.2 we've receiving this error in the Tyk logs for every our API:

[ERROR] trace: Failed to update baggage restrictions: StatusCode: 500, Body: collector error: baggage not implemented

We see this error for each API file every minute so its quite annoying. This error does not occur on a local environment on Docker Compose where we use jaegertracing/all-in-one:latest despite that request to http://jaeger:5778/baggageRestrictions?service=our-fancy-service-open-api from a Tyk container gives a 500 error response as in k8s cluster. We use a same tyk.conf file for a local development.

I'm creating an issue here because we use opentracing-go, jaeger-client-go and jaeger in other services with a same cluster configuration and have not seen this error elsewhere - only in the Tyk container.

Also, I need to mention that in the cluster before the error above (which repeats every minute for each API file) there is a slightly other error message:

[ERROR] trace: Failed to update baggage restrictions: Get "http://localhost:5778/baggageRestrictions?service=our-fancy-service-open-api": dial tcp [::1]:5778: connect: connection refused

Reproduction steps Steps to reproduce the behavior:

1. Enable tracing support as described here https://tyk.io/docs/advanced-configuration/distributed-tracing/jaeger/
2. Deploy Tyk of version 3.2.2 to a k8s cluster with a jaeger-agent (official image 1.22.0) as a sidecar container and other pod with jaeger (all-in-one:1.22.0).

Actual behavior See error [ERROR] trace: Failed to update baggage restrictions: StatusCode: 500, Body: collector error: baggage not implemented every minute for each API config file.

Expected behavior No error.

Screenshots/Video Believe me :)

Logs (debug mode or log file): Error messages above.

Configuration (tyk config file): We've tried a several configurations with a same error result:

    "tracing": {
        "enabled": true,
        "name": "jaeger",
        "options": {}
    }

    "tracing": {
        "enabled": true,
        "name": "jaeger",
        "options": {
            "baggage_restrictions": null
        }
    }

And finished with this one to suppress the error:

    "tracing": {
        "enabled": true,
        "name": "jaeger",
        "options": {
            "baggage_restrictions": {
                "denyBaggageOnInitializationFailure": true,
                "refreshInterval": "24h"
            }
        }
    }

Additional context Add any other context about the problem here.

[FredyR4zox]

Same, but my error is different: 2022/02/19 17:00:45 [ERROR] trace: Failed to update baggage restrictions: Get "http://localhost:5778/baggageRestrictions?service=dev-portal": dial tcp [::1]:5778: connect: connection refused

[vverbani]

Thanks for logging this. I'll relay this to our internal engineers for more information on what could be causing this error

[briangardner]

@vverbani - is there any follow up on this? I'm getting the same error.

[andyo-tyk]

Sorry for the gap in replies - we have been busy working on a replacement for the current open tracing solution (that relies on the Jaeger client libraries that are officially deprecated).

Do you still see this problem?

Open Telemetry is coming to Tyk soon and will provide you with a vastly superior distributed tracing solution.

[andyo-tyk]

We've launched support for OpenTelemetry with Tyk 5.2 and are deprecating support for Open Tracing.

My colleague Sonja wrote this blog back in December on the topic of migrating from Open Tracing to Open Telemetry.

This page in our docs provides guidance on how to set up Open Telemetry with Jaeger.

I recommend that you give this a try, as it will give you a significantly improved (and actively supported) experience.