[Bug]: Error sending data to New Relic OTEL service

[migueluzcategui]

What happened?

I have this code in my main.rs with Tokyo.

  set_max_level(LevelFilter::Debug);
    let mut map = tonic::metadata::MetadataMap::with_capacity(1);
    map.insert("api-key", config.new_relic_license_key.parse().unwrap());

    let tracer = opentelemetry_otlp::new_pipeline()
        .tracing()
        .with_exporter(
            opentelemetry_otlp::new_exporter()
                .tonic()
                .with_endpoint("https://otlp.eu01.nr-data.net:4317")
                .with_protocol(opentelemetry_otlp::Protocol::Grpc)
                .with_timeout(Duration::from_secs(3))
                .with_compression(Compression::Gzip)
                .with_metadata(map),
        )
        .with_trace_config(
            trace::config()
                .with_sampler(Sampler::AlwaysOn)
                .with_id_generator(RandomIdGenerator::default())
                .with_max_events_per_span(64)
                .with_max_attributes_per_span(16)
                .with_resource(Resource::new(vec![KeyValue::new(
                    "service.name",
                    format!("service-name-{}", config.environment),
                )])),
        )
        .install_batch(opentelemetry_sdk::runtime::Tokio)?;

    let collector = tracing_subscriber::registry()
        .with(tracing_opentelemetry::layer().with_tracer(tracer))
        // .with(tracing_opentelemetry::layer().with_meter(meter))
        .with(tracing_subscriber::fmt::layer());
    tracing::subscriber::set_global_default(collector)?;

tracing::debug!("listening on {}", addr);

The problem is that I am receiving this error as soon I start generating logs.

TRACE h2::proto::streams::counts: transition_after; stream=StreamId(1); state=State { inner: Closed(Error(GoAway(b"", FRAME_SIZE_ERROR, Library))) }; is_closed=true; pending_send_empty=true; buffered_send_data=0; num_recv=0; num_send=0
2024-03-06T13:06:11.546551Z DEBUG hyper::proto::h2::client: client response error: connection error detected: frame with invalid size
OpenTelemetry trace error occurred. Exporter otlp encountered the following error(s): the grpc server returns error (Unknown error): , detailed error message: h2 protocol error: http2 error: connection error detected: frame with invalid size

The error mentions something about SIZE, but I encounter the same issue if I change the port, the header name for the API key, or use an invalid API key value, an invalid URL, it doesn't matter there is always the error with the size. Additionally, adding encryption and reducing the size of the event batches didn't help either. Therefore, I believe this issue is unrelated to the size of the traces I'm sending.

I also noticed that changing the port to 4318 and using HTTP as protocol it doesn't do anything, the logs still mentioned this: the grpc server returns error.... I am doing something wrong?

This is my cargo.toml

opentelemetry-otlp = { version = "0.15.0", features = ["metrics", "logs", "gzip-tonic"]}
opentelemetry_sdk = { version = "0.22.1", features = ["rt-tokio"]}
opentelemetry = { version = "0.22.0", features = []}
tracing-opentelemetry = "0.23.0"

API Version

0.22.0

SDK Version

0.22.1

What Exporters are you seeing the problem on?

OTLP

Relevant log output

TRACE h2::proto::streams::counts: transition_after; stream=StreamId(1); state=State { inner: Closed(Error(GoAway(b"", FRAME_SIZE_ERROR, Library))) }; is_closed=true; pending_send_empty=true; buffered_send_data=0; num_recv=0; num_send=0
2024-03-06T13:06:11.546551Z DEBUG hyper::proto::h2::client: client response error: connection error detected: frame with invalid size
OpenTelemetry trace error occurred. Exporter otlp encountered the following error(s): the grpc server returns error (Unknown error): , detailed error message: h2 protocol error: http2 error: connection error detected: frame with invalid size