Datadog exporter does not work with async http client

[ArtBlnd]

OpenTelemetry trace error occurred. cannot send span to the batch span processor because the channel is closed

Default and custom client both does not work. same error message occurs.

#[derive(Debug)]
struct ReqwestClient {
    client: reqwest::Client
}

#[async_trait::async_trait]
impl HttpClient for ReqwestClient {
    async fn send(&self, request: Request<Vec<u8>>) -> Result<Response<Bytes>, HttpError> {
        Ok(self.client.send(request).await?)
    }
}

[TommyCpp]

It may not be a problem with datadog exporter. Could you share a more detailed example of how you configure the tracer provider? Or maybe try using some other exporters and see if it works. It may help us narrow down the cause. Thanks in advance.

[ArtBlnd]

let tracer = new_pipeline()
            .with_trace_config(trace::config().with_sampler(Sampler::AlwaysOn))
            .with_service_name("test")
            .with_version(ApiVersion::Version05)
            .with_agent_endpoint("localhost")
            .with_http_client::<ReqwestClient>(Box::new(ReqwestClient {
                client: reqwest::Client::new(),
            }))
            .install_batch(Tokio)
            .expect("failed to initialize tracing pipeline");

        tracing_subscriber::fmt()
            .with_ansi(false)
            .with_writer(std::io::stderr)
            .with_target(false)
            .finish()
            .with(OpenTelemetryLayer::new(tracer))
            .init();

[ArtBlnd]

It does work fine with blocking reqwest http client. but emit error sometimes.

OpenTelemetry trace error occurred. error sending request for url (http://localhost/v0.5/traces): connection closed before message completed

[ArtBlnd]

If I unwrap error. I got this error from reqwest

IoError(Custom { kind: Other, error: "Connection broken" })

[ArtBlnd]

opentelemetry-jaeger works fine.

[TommyCpp]

It feels like a communication problem with the datadog collector. I noticed you are sending spans to http://localhost. Is that expected?

[ArtBlnd]

Thats not a communication problem. that is expected. I changed it to localhost on text not to expose external ip address. (in the real code uses actual address with working port.)

surely its datadog exporter bug.

The exporter does not work with ASYNC http client. it works fine with BLOCKING REQWEST http client. which I already notice.

[TommyCpp]

Thanks for the bug report. I did try to reproduce this bug locally. Here is my current setup.

Here is my approach:

You mentioned the default reqwest async client is not working so I enabled the reqwest-client feature, which should provide a default async reqwest client. Then I start a datadog agent container locally. Running the example and validate the span is show up in the datadog APM console.

I think I am missing something here, could you take a look and see if there is anything that need to change in my example?

[rex-remind101]

fwiw, I ran into this exact same problem last week.

[rex-remind101]

surf-client seems to work until you try install_batch https://github.com/open-telemetry/opentelemetry-rust/issues/724

[rex-remind101]

I came to the realization that when using reqwest-client with install_batch I do receive traces in datadog. However, I had set the primary operation name to one of thee child spans operation names and not the root span, and when using install_batch the child spans do not seem to have individual traces. When I set the UI configuration in DataDog to use the root as the primary operation name I do see traces however. This issue does not happen when using surf-client + install_simple, I see all traces in that case. (I set the primary operation name to a child for required reasons I won't dive into, but this is blocking me because I need this configured as such.)

I'm not sure if this will help your case, but I can at least confirm this batch + reqwest-client "works" but is not doing everything I'd expect.

[rex-remind101]

And surf-client does not work with install_batch still.

[alexpusch]

I'm getting OpenTelemetry trace error occurred. error sending request for url (http://<ip>:8126/v0.5/traces): connection closed before message completed as well. Using reqwest-client and install_batch. Traces do reach Datadog, so it seems that this error is not critical, just annoying. I do not see any error on the datadog trace agnet side