[Go] Genkit overrides global OTel TracerProvider after Init/flow, breaking Pyroscope instrumentation

[pavhl]

Describe the bug Genkit overrides the global OpenTelemetry tracer provider when the host application uses Pyroscope's otel-profiling-go wrapper provider. After calling genkit.Init and running a flow, otel.GetTracerProvider() changes from the Pyroscope wrapper to a plain SDK provider, which disables Pyroscope instrumentation.

Observed output from the minimal repro below:

Before genkit: *otelpyroscope.tracerProvider (0x40001910e0)
After genkit: *trace.TracerProvider (0x4000146630)

This indicates Genkit replaced the global provider that had been set to the Pyroscope wrapper.

To Reproduce

1. Use the following minimal program (Go) which sets up an SDK tracer provider, wraps it with Pyroscope's provider, then runs a genkit flow and compares before/after types of otel.GetTracerProvider().

package main

import (
    "context"
    "errors"
    "fmt"
    "io"
    "log"
    "time"

    "github.com/firebase/genkit/go/genkit"
    otelpyroscope "github.com/grafana/otel-profiling-go"
    "github.com/grafana/pyroscope-go"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/stdout/stdouttrace"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    "go.opentelemetry.io/otel/trace"
)

func main() {
    ctx := context.Background()

    tracerProvider, shutdown, err := initTracer(ctx)
    if err != nil {
        log.Fatalf("Failed to initialize tracer: %v", err)
    }
    defer shutdown(ctx)

    // Wrap SDK provider with Pyroscope provider
    otel.SetTracerProvider(otelpyroscope.NewTracerProvider(tracerProvider))

    _, err = pyroscope.Start(pyroscope.Config{
        ApplicationName: "my-service",
        ServerAddress:   "http://localhost:4040",
    })
    if err != nil {
        log.Fatalf("Failed to initialize pyroscope: %v", err)
    }

    fmt.Printf("Before genkit: %T (%p)\n", otel.GetTracerProvider(), otel.GetTracerProvider())

    gkit := genkit.Init(ctx)
    testFlow := genkit.DefineFlow(gkit, "test", func(ctx context.Context, input string) (string, error) {
        return input, nil
    })
    testFlow.Run(ctx, "foo")

    fmt.Printf("After genkit: %T (%p)\n", otel.GetTracerProvider(), otel.GetTracerProvider())
}

func initTracer(ctx context.Context) (trace.TracerProvider, func(context.Context) error, error) {
    var shutdownFuncs []func(context.Context) error
    var err error

    shutdown := func(ctx context.Context) error {
        var err error
        for _, fn := range shutdownFuncs {
            err = errors.Join(err, fn(ctx))
        }
        shutdownFuncs = nil
        return err
    }

    handleErr := func(inErr error) {
        err = errors.Join(inErr, shutdown(ctx))
    }

    tracerProvider, err := newTracerProvider()
    if err != nil {
        handleErr(err)
        return tracerProvider, shutdown, err
    }
    shutdownFuncs = append(shutdownFuncs, tracerProvider.Shutdown)
    otel.SetTracerProvider(tracerProvider)

    return tracerProvider, shutdown, err
}

func newTracerProvider() (*sdktrace.TracerProvider, error) {
    traceExporter, err := stdouttrace.New(stdouttrace.WithWriter(io.Discard))
    if err != nil {
        return nil, err
    }
    tracerProvider := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(traceExporter, sdktrace.WithBatchTimeout(time.Second)),
    )
    return tracerProvider, nil
}

2. Run:

export GENKIT_ENV=dev && genkit ui:start && genkit start -- go run main.go

3. Observe that the global provider type has changed away from the Pyroscope wrapper.

Expected behavior

• Genkit should not override a pre-configured global tracer provider.
• At minimum, if the current provider is not an SDK provider, Genkit should avoid replacing it and document how to integrate exporters.
• Genkit UI should work even if the host application has its own OpenTelemetry setup.

Screenshots N/A (see console output above).

Runtime (please complete the following information):

• OS: macOS
• Version: 15.7.1

Go version

• go version go1.25.2 darwin/arm64

Additional context

• Module versions:

  • github.com/firebase/genkit/go v1.0.5
  • github.com/grafana/otel-profiling-go v0.5.1
  • github.com/grafana/pyroscope-go v1.2.7
  • go.opentelemetry.io/otel v1.38.0
  • go.opentelemetry.io/otel/sdk v1.38.0
  • go.opentelemetry.io/otel/trace v1.38.0
  • go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.38.0

• Likely cause in Genkit (v1.0.5): core/tracing/tracing.go unconditionally sets a new SDK provider if the current provider is not exactly *sdktrace.TracerProvider, then returns it by asserting the global provider back to *sdktrace.TracerProvider:

func TracerProvider() *sdktrace.TracerProvider {
    if tp := otel.GetTracerProvider(); tp != nil {
        if sdkTP, ok := tp.(*sdktrace.TracerProvider); ok {
            return sdkTP
        }
    }

    providerInitOnce.Do(func() {
        otel.SetTracerProvider(sdktrace.NewTracerProvider())
        if telemetryURL := os.Getenv("GENKIT_TELEMETRY_SERVER"); telemetryURL != "" {
            WriteTelemetryImmediate(NewHTTPTelemetryClient(telemetryURL))
        }
    })

    return otel.GetTracerProvider().(*sdktrace.TracerProvider)
}

• This replaces wrapper providers such as otelpyroscope.tracerProvider, and the final type assertion can panic if a non-SDK provider is still installed.

[pavhl]

For now we forked the library and used this as a workaround which fixes our usecase: https://github.com/pavhl/genkit/commit/a7eecb936c97ef0578d1c330420978a061792eec#diff-d8bc9bb430176a624763f21c1bcd2ad7ee40e9f475b18f107529e677a7484136

The change:

• ensures that the genkit dev UI tracing works independently from the global OTel instance by passing it via context in reflection handler
• prevents genkit from replacing custom global TracerProvider

But I'm not sure if it breaks other usecases (e.g. when using cloud tracing)

[eddyleelin]

For reference, this is super similar to the same problems we had with Sentry, as described in #2904. Although Sentry luckily exposes a way to use a custom OpenTelemetry provider (so we can use Genkit's initialized one)