Autoconfigurable Telemetry SDK support

[Shephalimittal]

Description

Added support for OpenTelemetry SDK Autoconfigure. This artifact implements environment-based autoconfiguration of the OpenTelemetry SDK. This can be an alternative to programmatic configuration using the normal SDK builders.

All options support being passed as Java system properties, e.g., -Dotel.traces.exporter=zipkin or environment variables, e.g., OTEL_TRACES_EXPORTER=zipkin. For more details refer here.

The intention here is to remove managing tracing configuration like exporter, endpoints, processor delays, sampler etc through Opensearch cluster settings. Also, if there are any custom exporter or sampler settings, user can pass that through jvm options or environment variables.

This also provides support to add custom exporters. InMemorySpanExporter is the exporter used for integration tests, have made it autoconfigurable by using ConfigurableSpanExporterProvider class

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• [ ] New functionality includes testing.
  • [ ] All tests pass
• [ ] New functionality has been documented.
  • [ ] New functionality has javadoc added
• [ ] Commits are signed per the DCO using --signoff
• [ ] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/23088/
• CommitID: 4df8085f952df8ee43b29f9ac74c1d2afcab1f0e Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/23193/
• CommitID: 7270f44ac68ddbed65a3892817bec5bfb5f3a329 Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/23195/
• CommitID: ec1a16fc4f6a87627c2fe7f23669a72505081922 Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[opensearch-trigger-bot[bot]]

Compatibility status:

Checks if related components are compatible with change dd75a22

Incompatible components

Incompatible components: [https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/security-analytics.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

[Shephalimittal]

@reta Please review.

[reta]

@reta Please review.

@Shephalimittal please, first of all, fill the description for this change, so everyone would understand what is the purpose of it. Please also add the changelog entry under [Unreleased 2.x] section, thank you.

[reta]

@Shephalimittal OTel auto configuration is certainly useful for application developers, now I am struggling to understand how that will work in OpenSeach:

• user installs OpenSeach
• user install OpenSearch Otel plugin

User wants to configure OpenSearch Otel plugin to send traces to Jaeger (for example), how that could be accomplished with auto configuration?

[Gaganjuneja]

@Shephalimittal OTel auto configuration is certainly useful for application developers, now I am struggling to understand how that will work in OpenSeach:

• user installs OpenSeach
• user install OpenSearch Otel plugin

User wants to configure OpenSearch Otel plugin to send traces to Jaeger (for example), how that could be accomplished with auto configuration?

As per my understanding, user will have to provide the jaeger related configurations in the jvm.options file.

[reta]

As per my understanding, user will have to provide the jaeger related configurations in the jvm.options file.

If that is the case (which could only be done for some properties), this is certainly no go: we have established settings management.

[Gaganjuneja]

As per my understanding, user will have to provide the jaeger related configurations in the jvm.options file.

If that is the case (which could only be done for some properties), this is certainly no go: we have established settings management.

Only challenge with doing setting management ourselves is we always need to sync with the OTEL. The OTLPGrpcSpanExporter itself has more than 10 settings and similarly BatchSpanProcessor. So we need to expose these all through OpenSearch settings and if there are new types of exporters we might have to add support.

OTelAutoConfigureSDK reads the settings/configuration from System properties so not sure if opensearch.yml can also be used for the same.

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/23516/
• CommitID: 86d2656d1b16bc5a990be8af3afaa6119f519198 Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/23517/
• CommitID: 1e54dbd2b4b2e8a198b712df6e6d3be345a06682 Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[Shephalimittal]

@reta Please review.

[reta]

@reta Please review.

@Shephalimittal please answer my questions (https://github.com/opensearch-project/OpenSearch/pull/9453#issuecomment-1690182246)

[Shephalimittal]

@Shephalimittal OTel auto configuration is certainly useful for application developers, now I am struggling to understand how that will work in OpenSeach:

• user installs OpenSeach
• user install OpenSearch Otel plugin

User wants to configure OpenSearch Otel plugin to send traces to Jaeger (for example), how that could be accomplished with auto configuration?

Yes, that will be achieved through this setting -Dotel.traces.exporter=jaeger in jvm.options. For details refer link

Also, if user wants to set up a custom exporter, that can be provided using implementing ConfigurableSpanExporterProvider. I have added InMemorySingletonSpanExporterProvider for integration test.

[reta]

Thanks @Shephalimittal

Yes, that will be achieved through this setting -Dotel.traces.exporter=jaeger in jvm.options. For details refer link

We use jvm.options to configure the JVM parameters for JVM process, not plugins / module or extensions: we use opensearch.yaml for configuration of deployment specific setting and this is what the otel-plugin should also do.

Also, if user wants to set up a custom exporter, that can be provided using implementing ConfigurableSpanExporterProvider. I have added InMemorySingletonSpanExporterProvider for integration test.

Very well, but in most cases the additional dependency (primarily, exporters) have to be provided (and we cannot bundle everything), so it brings us to the the following question: how user should do that? Manually copying JAR files does not look like a viable option at all (since those may require additional dependencies).

[Shephalimittal]

Only challenge with doing setting management ourselves is we always need to sync with the OTEL. The OTLPGrpcSpanExporter itself has more than 10 settings and similarly BatchSpanProcessor. So we need to expose these all through OpenSearch settings and if there are new types of exporters we might have to add support.

Thanks @Shephalimittal

Yes, that will be achieved through this setting -Dotel.traces.exporter=jaeger in jvm.options. For details refer link

We use jvm.options to configure the JVM parameters for JVM process, not plugins / module or extensions: we use opensearch.yaml for configuration of deployment specific setting and this is what the otel-plugin should also do. But problem with using cluster settings is : (Pasting Gagan's point from above) Only challenge with doing setting management ourselves is we always need to sync with the OTEL. The OTLPGrpcSpanExporter itself has more than 10 settings and similarly BatchSpanProcessor. So we need to expose these all through OpenSearch settings and if there are new types of exporters we might have to add support.

Also, if user wants to set up a custom exporter, that can be provided using implementing ConfigurableSpanExporterProvider. I have added InMemorySingletonSpanExporterProvider for integration test.

Very well, but in most cases the additional dependency (primarily, exporters) have to be provided (and we cannot bundle everything), so it brings us to the the following question: how user should do that? Manually copying JAR files does not look like a viable option at all (since those may require additional dependencies).

[Shephalimittal]

Only challenge with doing setting management ourselves is we always need to sync with the OTEL. The OTLPGrpcSpanExporter itself has more than 10 settings and similarly BatchSpanProcessor. So we need to expose these all through OpenSearch settings and if there are new types of exporters we might have to add support.

Thanks @Shephalimittal

Yes, that will be achieved through this setting -Dotel.traces.exporter=jaeger in jvm.options. For details refer link

We use jvm.options to configure the JVM parameters for JVM process, not plugins / module or extensions: we use opensearch.yaml for configuration of deployment specific setting and this is what the otel-plugin should also do. But problem with using cluster settings is : (Pasting Gagan's point from above) Only challenge with doing setting management ourselves is we always need to sync with the OTEL. The OTLPGrpcSpanExporter itself has more than 10 settings and similarly BatchSpanProcessor. So we need to expose these all through OpenSearch settings and if there are new types of exporters we might have to add support.

Also, if user wants to set up a custom exporter, that can be provided using implementing ConfigurableSpanExporterProvider. I have added InMemorySingletonSpanExporterProvider for integration test.

Very well, but in most cases the additional dependency (primarily, exporters) have to be provided (and we cannot bundle everything), so it brings us to the the following question: how user should do that? Manually copying JAR files does not look like a viable option at all (since those may require additional dependencies).

Exploring the option to use AffixSettings here for telemetry settings regarding exporters, delays, timeouts, samplers etc and passing these settings as configProperties to AutoConfiguredOpenTelemetrySdk. @reta Any Thoughts?

[reta]

Exploring the option to use AffixSettings here for telemetry settings regarding exporters, delays, timeouts, samplers etc and passing these settings as configProperties to AutoConfiguredOpenTelemetrySdk. @reta Any Thoughts?

@Shephalimittal we need more than that:

• dependencies (new exporters and samplers)
• security policy changes

In general, I think we have to take a step back and open RFC issue to outline the vision regarding extensibiltity / customization of the out OTEL plugin (I personally don't see the large picture).

[opensearch-trigger-bot[bot]]

This PR is stalled because it has been open for 30 days with no activity.

[ticheng-aws]

Hi @Shephalimittal, the PR is stalled. Is this being worked upon? Feel free to reach out to maintainers for further reviews.

[opensearch-trigger-bot[bot]]

This PR is stalled because it has been open for 30 days with no activity.

[kotwanikunal]

@Shephalimittal Closing this stalled PR. Please reopen to continue with the contribution.