Tracking: Stabilize declarative configuration

[jack-berg]

We've made a lot of progress in declarative configuration, and I'd like to discuss what scope should be included in an initial cut at stabilization, and blockers.

The declarative config spec is broken into a variety of pieces:

• Data model: SDK implementation requirements around data model, and definition of schema. Subcomponents:
  • opentelemetry-configuration: repository for data model JSON schema
  • file-based YAML representation, including env var substitution syntax
  • SDK in-memory representation
• Instrumentation configuration API: defines an API for instrumentation to participate in configuration. Subcomponents:
  • ConfigProperties for programmatic access of structured config
  • ConfigProvider for accessing instrumentation config
• SDK: SDK user facing capabilities. Subcomponents:
  • ConfigProvider SDK implementation of ConfigProvider API component
  • ComponentProvider defines mechanism for referencing / loading SDK extension plugin components (i.e. exporters, samplers, etc) in declarative config
  • Parse and Create operations for producing configured SDK components from YAML file representation
• OTEL_EXPERIMENTAL_CONFIG_FILE, for automatic initialization based on YAML file representation

I propose the following scope for initial stability:

• opentelemetry-configuration
• file-based YAML representation
• SDK in-memory representation
• ConfigProperties
• ComponentProvider
• Parse and Create operations
• OTEL_EXPERIMENTAL_CONFIG_FILE, adjusted to OTEL_CONFIG_FILE

That means the following would be out of scope:

• ConfigProvider (i.e. instrumentation config API)

Assuming this scope, the follow lists the known blockers to stability

• [ ] https://github.com/open-telemetry/opentelemetry-configuration/issues/161 (see list of sub-issues)
• [x] #4027
• [x] #3963
  • [x] #3914

The following list are not specifically blocking, but they should be reviewed to ensure the stability of declarative configuration doesn't prevent us from addressing them in the future:

Additionally, we need evaluation / feedback of the state of the spec from the current prototype language implementations.

If you have feedback on the scope of initial stability, or on additional blockers, please discuss in this issue.

[codeboten]

Not strictly blocking, but we should ensure we're not preventing addressing issues like this one https://github.com/open-telemetry/opentelemetry-specification/issues/2739

[pellared]

Please consider: https://github.com/open-telemetry/opentelemetry-specification/issues/4384

I was also thinking if the declarative configurations should allow adding custom components e.g. custom span processor created by the user. Right now it looks like the declarative configuration can be used only in a sandbox.

[jack-berg]

FYI, for anyone following this issue, the issue #4384 linked above has been closed. Please see the issue for details.

[jack-berg]

#4416 was opened and its worth discussing whether its blocking for stability. Currently, the declarative config spec accommodates customization via language in OTEL_EXPERIMENTAL_CONFIG_FILE:

Implementations MAY provide a mechanism to customize the configuration model parsed from OTEL_EXPERIMENTAL_CONFIG_FILE.

My opinion is that this language is flexible enough for opentelemetry-java and any other implementation that has customization requirements to build out tooling. Although, as a maintainer of opentelemetry-java, I would hesitate to make such tooling stable without more discussion in the spec, since if there is going to be more standardization around how customization works I would want opentelemetry-java to conform to that.

I personally don't consider #4416 blocking since: 1. Declarative config is still quite useful without it. 2. There is nothing in the spec that prevents future enhancement in this area. I do think #4416 is a very interesting area as a followup to stabilization.

Interested in what others think, including @open-telemetry/configuration-approvers.

[jack-berg]

FYI, removing #2739 from list of potential blockers per this comment.

[jack-berg]

Update on the status of this:

All the spec issues and opentelemetyr-configuration issues identified for a stable release (#161) are complete.

As we applied additional scrutiny to the schema in preparation for stability, we made a decent number of breaking changes in an effort to be more correct and consistent. The current plan is:

1. cut a 0.4.0 minor release
2. let the dust settle, uncover / fix any issues as more implementations catch up
3. cut 1.0.0-rc release candidate, uncover / fix any issues
4. cut a 1.0.0 stable release

[pellared]

Shouldn't https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/sdk_exporters/prometheus.md be stabilized first as the configuration options for Prometheus exporter may change? For example see: https://github.com/open-telemetry/opentelemetry-specification/issues/4223#issuecomment-2839595942

[jack-berg]

The declarative config data model has mechanisms to allow types / properties to be marked as experimental: https://github.com/open-telemetry/opentelemetry-configuration/tree/main?tab=readme-ov-file#experimental-features

Because the prometheus exporter specification isn't stable, the promeheus exporter type in the declarative config schema is marked experimental: https://github.com/open-telemetry/opentelemetry-configuration/blob/main/schema/meter_provider.json#L188-L211

[marcalff]

@jack-berg Possibly, add #3967 to the list of known blockers ?

[marcalff]

@jack-berg

Also, please take a look at #4594 . The issue there is whether or not to allow yet another new environment variable, that does not fit directly into declarative configuration.

The issue is mostly about scheduling, it is hard to:

• forbid new env vars
• mandate to use declarative configuration instead

when declarative configuration is not available (and sometime not even planned) for a every SIG SDK.