opentelemetry-rust icon indicating copy to clipboard operation
opentelemetry-rust copied to clipboard

[Bug]: OpenTelemetry log error occurred. cannot send message to batch processor as the channel is closed

Open NickLarsenNZ opened this issue 10 months ago • 4 comments

What happened?

When Tokio shuts down, I receive a lot of errors about a channel being closed: This happens whether or not I attempt to shutdown (unset) the trace/log providers.

Here is a brief explanation as code:

#[tokio::main]
async fn main() {
    // initialise tracing/logging here

    opentelemetry::global::shutdown_tracer_provider();
    opentelemetry::global::shutdown_logger_provider();

    // Without this sleep, I get a different error:
    // OpenTelemetry log error occurred. Exporter otlp encountered the following errors: the grpc server returns error (Unknown error): , detailed error message: transport error
    tokio::time::sleep(Duration::from_secs(2)).await;

    // errors are printed at this point.
}

For a full (hopefully) reproducable version of the code and dependencies, see: https://gist.github.com/NickLarsenNZ/fbbb477230e9992e9a4aa2603edcf54f

I initially raised it in the #otel-rust Slack channel: https://cloud-native.slack.com/archives/C03GDP0H023/p1712590806651679

API Version

0.22.0

SDK Version

0.22.1

What Exporters are you seeing the problem on?

OTLP

Relevant log output

OpenTelemetry log error occurred. cannot send message to batch processor as the channel is closed.

NickLarsenNZ avatar Apr 11 '24 15:04 NickLarsenNZ

The batch log processor has a background task that exporting the logs. When it's being droped the receiver in the background task will gets dropped too.

After this operation, every attempt to emit logs will result in such error.

TommyCpp avatar Apr 15 '24 00:04 TommyCpp

I am seeing this too. Is it safe not to call opentelemetry::global::shutdown_tracer_provider(); when the application exits to avoid this error? What are the other implications of doing so?

ramgdev avatar Aug 20 '24 17:08 ramgdev

so if we have this error, it means that the tracer provider has not been shutdown correctly before the application stopped?

I have this error even when I call opentelemetry::global::shutdown_tracer_provider() before ending my application, is it an expected behavior?

Oliboy50 avatar Oct 18 '24 17:10 Oliboy50

on my side, I realized that this error occurred after cloning the pipeline provider with something like:

let provider = otlp_pipeline
    .install_batch(opentelemetry_sdk::runtime::Tokio)
    .expect("Failed to build OTLP pipeline")
    .provider()
    .clone();

and then running opentelemetry::global::shutdown_tracer_provider(); would not shutdown everything (and therefore would gave me the error mentioned above) if the program ends while provider has not also been dropped

Oliboy50 avatar Oct 21 '24 13:10 Oliboy50