Specifying tracing layer twice with different configuration and filters results in panic

[inikulin]

Bug Report

Version

tracing-experiments v0.1.0 (/Users/inikulin/Work/tracing-experiments)
├── tracing v0.1.35
│   ├── tracing-attributes v0.1.21 (proc-macro)
│   └── tracing-core v0.1.27
├── tracing-opentelemetry v0.17.3
│   ├── tracing v0.1.35 (*)
│   ├── tracing-core v0.1.27 (*)
│   ├── tracing-log v0.1.3
│   │   └── tracing-core v0.1.27 (*)
│   └── tracing-subscriber v0.3.11
│       ├── tracing-core v0.1.27 (*)
│       └── tracing-log v0.1.3 (*)
└── tracing-subscriber v0.3.11 (*)

Platform

Darwin C02FF11XMD6R 21.5.0 Darwin Kernel Version 21.5.0: Tue Apr 26 21:08:22 PDT 2022; root:xnu-8020.121.3~4/RELEASE_X86_64 x86_64

Description

I'm trying to have 2 opentelemetry tracing layers in my app. Layers have different configuration and spans dispatched to them via filters based on attributes. However, calling tracing_subscriber::registry().with_tracer() twice with opentelemetry tracing layer results in panic:

     Running `target/debug/tracing-experiments`
thread 'main' panicked at 'assertion failed: self.replace(val).is_none()', /Users/inikulin/.cargo/registry/src/github.com-1ecc6299db9ec823/tracing-subscriber-0.3.11/src/registry/extensions.rs:88:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[davidbarsky]

The issue is that tracing-subscriber's Registry is a typemap for extensions and tracing-opentelemetry does not create distinct types that vary between different Layer instantiations.

(can you also edit your bug report to include the causing the crash? It makes debugging/searching for issues easier.)

[penso]

Same issue here when trying to send traces to 2 different services to "benchmark" them (jaeger backend and otlp backend).

[gliderkite]

I'm following the example found in the opentelemetry-rust project. If I try to add a new layer to the Registry I get a panic error message when I cURL the endpoint:

let tracer_datadog = opentelemetry_datadog::new_pipeline()
    .with_service_name("actix_http_tracing_datadog")
    .with_version(opentelemetry_datadog::ApiVersion::Version05)
    .install_batch(opentelemetry::runtime::Tokio)
    .unwrap();

Registry::default()
    .with(tracing_subscriber::EnvFilter::new("INFO"))
    .with(tracing_subscriber::fmt::layer())
    .with(tracing_opentelemetry::layer().with_tracer(tracer_jaeger))
    .with(tracing_opentelemetry::layer().with_tracer(tracer_datadog))
    .init();

thread 'actix-rt|system:0|arbiter:0' panicked at 'assertion failed: self.replace(val).is_none()', ~/.cargo/registry/src/github.com-1ecc6299db9ec823/tracing-subscriber-0.3.16/src/registry/extensions.rs:88:9

[James-Bartman]

So is there any way to send data in OTLP format to 2 different places (ie. a backend and write to a file)? I'm trying to use opentelemetry_otlp and opentelemetry_stdout but can't find any way to do that without running into this error.

[dotansimha]

@inikulin did you manage to find a solution for that? I'm also trying to register multiple filter layers with different outputs, and it seems to panic because of the same reason.