Remote Configuration for Traces/Metrics GRPC

[zalseryani]

Is your feature request related to a problem? Please describe

The feature is required to send traces and metrics of opensearch to external monitoring tools like signoz.

• We have tested metrics and traces, it's working properly on a log file, and tested the grpc, and working fine, but we were not able to use a specific configuration for otlp to send those traces and metrics to signoz as an example.

• We checked the docs and noticed that it's not configurable.

• We found a typo related to the grpc configuration for traces, the class in the docs is mentioned incorrectly telemetry.otel.tracer.span.exporter.class=org.opensearch.telemetry.tracing.exporter.OtlpGrpcSpanExporterProvider, while it should have the following value telemetry.otel.tracer.span.exporter.class: io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter

• Another note would be related to trace configuration, telemetry.feature.tracer.enabled: true this configuration is not mentioned in the docs, I got the support from @Gaganjuneja in another story Metrics Framework, and tested and it's working fine for the trace file log configuration.

Describe the solution you'd like

• There are 3 solutions 1- Create a new environment variable for this case. 2- Reuse the environment variable featured by opentelemetry OTEL_EXPORTER_OTLP_ENDPOINT. 3- Create a configmap or additional opensearch.yml configuration.

Related component

Plugins

Describe alternatives you've considered

The alternative solution now is to use a reverse proxy for rerouting / redirecting traffic to a remote otel-collector server, but I believe it's not practical and I would not use it within Kubernetes cluster deployment.

Additional context

No response

[peternied]

[Triage - attendees 1 2 3 4 5 6 7] @zalseryani Thanks for creating this issue, note this is a little high level - it isn't exactly clear how this would happen or if there is good patterns for reuse elsewhere in OpenSearch. Pull requests might see some extra design discussion that would be good to head off with an RFC- for your consideration.

[Gaganjuneja]

@zalseryani thanks for opening this issue.

1. For distributed tracing documentation fix, I have created an issue and will address that.
2. To support configuring the exported end point, we already had discussion and I need to revive this thread with the best possible option. issue - https://github.com/opensearch-project/OpenSearch/issues/9764