trafficserver [PLugin] OpenTelemetry Tracer Plugin make request processing time increase

Hi everyone, I using TrafficSrever branch: master - #828b7d3. I trying OpenTelemetry Tracer Plugin to manage tracer for system. When i use the plugin, it make request processing time increase.

When use plugin: has_otel req_has_otel

Don't use plugin: no_otel req_no_otel

Has anyone used this plugin yet?

Jul 06 '23 05:07 HugManh

How does your observability stack look like?

The plugin is sending a OTLP http request for sending the trace info to a backend. So if your backend is quite far away, it can take quite some time.

One alternative is this - https://opentelemetry.io/docs/collector/deployment/agent/ You can have the deployment of an agent/collector on the same host of your ATS. And you configure the plugin to send OTLP request to the agent/collector instead. That should be much faster. And then you let the agent/collector do its work to send the info to a real backend.

Jul 06 '23 06:07 shukitchan

How does your observability stack look like?

The plugin is sending a OTLP http request for sending the trace info to a backend. So if your backend is quite far away, it can take quite some time.

My backend is pretty close so I think it's not due to this. I tried with opentelemetry-cpp version 1.3.0 and as a result request time is faster than branch master . Note: The results I posted above is from the opentelemetry branch master#. Do you think the problem is with the sdk? Result of version 1.3.0: photo_2023-07-06_22-00-08

One alternative is this - https://opentelemetry.io/docs/collector/deployment/agent/ You can have the deployment of an agent/collector on the same host of your ATS. And you configure the plugin to send OTLP request to the agent/collector instead. That should be much faster. And then you let the agent/collector do its work to send the info to a real backend.

I will try this later, Thankyou @shukitchan

Jul 06 '23 15:07 HugManh

300+ms is still a lot. I think we should try having the agent/collector in the local host.

I have not got the time to try the latest opentelemetry-cpp release. So I can't say if there is anything changed in the sdk that will cause this much latency.

Jul 06 '23 16:07 shukitchan

@shukitchan Have you tried using opentelemetry without collector?

Jul 07 '23 03:07 HugManh

I think i found problem in the plugin. I change span processor from SimpleSpanProcessor to BatchSpanProcessor. This solved my problem. @shukitchan Thank you very much!

Jul 07 '23 04:07 HugManh

What would be the number after the change?

Do you want to submit a pull request for the change?

Jul 10 '23 20:07 shukitchan

This issue has been automatically marked as stale because it has not had recent activity. Marking it stale to flag it for further consideration by the community.

Jul 10 '24 01:07 github-actions[bot]