Opentelemetry operator able to create and mount array of config / configmap [yaml's] instead of single configmap creation

[kabhanus007]

cc: @pavankrish123 @jpkrohling Hi Opentelemetry Operator team,

Background Information

1. Opentelemetry collector now supports multiple configuration https://github.com/open-telemetry/opentelemetry-collector/pull/4742
2. Currently opentelemetry operator takes single configuration yaml as spec.config for opentelemetry collector CRD
3. We would like to leverage point (1) above, and pass in configuration as multiple YAML objects.

Proposed Solution:

1. Instead of single spec.config for Opentelemetry collector CRD, we propose a new field, configYamls which is an array of yaml objects instead of single YAML.
2. These YAML objects will eventually be written as multiple YAML files in the otelcollector pod managed by opentelemetry operator

Sample Implementation:

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: simplest
spec:
  configYamls:
  - prometheusConf: |
     // yaml configuration inside which will be used to create prometheus configmap
  - otelBaseConf: | 
     // yaml configuration inside which will be used to create otelBaseConf configmap

Once the above configuration is applied, 2 files prometheusConf.yaml and otelBaseConf.yaml will be created in the pod and ./otelcorecol --config=file:prometheusConf.yaml --config=file:otelBaseConf.yaml needs to be executed as start cmd.

We use a helm chart to manage these CRD and we would like to leverage this multiple configuration feature to inject multiple configuration from multiple teams through separate helm template files. So far we have been using helm logic to merge these configurations, but it is getting complicated. With this implementation, we can completely eliminate the merge logic from our helm chart.

[Aneurysm9]

Beyond specifying multiple YAML configuration blocks, it might also be nice to be able to reference configuration sources by URI to allow loading config from other than a config map created by the operator.

[kabhanus007]

Beyond specifying multiple YAML configuration blocks, it might also be nice to be able to reference configuration sources by URI to allow loading config from other than a config map created by the operator.

@Aneurysm9 , maybe if you can add some example to elaborate your statement. But yes, that's a kind of another good suggestion to add on top of this.

[yuriolisa]

@kabhanus007, bringing the ideas together, I would suggest a third via, like:

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: simplest
spec:
  manifests: 
  - name:
     type: <Prometheus, Otel, etc..>
     url: <following the @Aneurysm9  suggestion >
     configMap: <use either url or configMap>

[kabhanus007]

@kabhanus007, bringing the ideas together, I would suggest a third via, like:

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: simplest
spec:
  manifests: 
  - name:
     type: <Prometheus, Otel, etc..>
     url: <following the @Aneurysm9  suggestion >
     configMap: <use either url or configMap>

thanks for suggesting and adding some extra useful stuff to it, it seems a good addition, but a question, how type parameter will be used and are we going to have some fixed constants for that?

[jpkrohling]

instead of multiple configs, I would rather prefer to let users provide either a single config block, or a set of config URIs. This way, the config file can still be simple for most use cases, while providing a construct for more complex use cases. In such case, users would be required to manage the external sources, including configmaps.

[kabhanus007]

instead of multiple configs, I would rather prefer to let users provide either a single config block, or a set of config URIs. This way, the config file can still be simple for most use cases, while providing a construct for more complex use cases. In such case, users would be required to manage the external sources, including configmaps.

Yeah, that should also work which will ultimately keep this simple config block the same, but having the capability of extending the functionality of working with multiple configmaps through complex configuration.

  apiVersion: opentelemetry.io/v1alpha1
  kind: OpenTelemetryCollector
  metadata:
    name: complex
  spec:
    uri:
    // configmaps location URL
    - path1
    - path2

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: simplest
spec:
  config: |
  // Simple Config Block

Is this something we are looking forward to it? @jpkrohling