Support for dynamic collector config

[RashmiRam]

Usecase/Feature/Problem Statement

Collector supports attribute processors and few exporters where the config requires to be dynamic based on the use case.

Current approach

Include the collector config as part of the lambda function and reference the file with env var.

Challenges in the current approach

• This requires changing the lambda function
• All config(ex., exporter with headers containing any auth token, referring to AWS secrets) can't be committed to SCM. Few has to reference env var or AWS secrets.

Proposal

I don't have a detailed proposal yet. Just wanted to check if this makes sense. Provision to add dynamic config with some kind of templating to generate the config when collector is instantiated.

[rijulg]

We came across this problem as well while trying to reconfigure the collector to use the OTLP exporter with other vendors. Our configuration looks like the following:

#collector.yaml in the root directory
#Set an environemnt variable 'OPENTELEMETRY_COLLECTOR_CONFIG_FILE' to '/var/task/collector.yaml'

receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  logging:
    loglevel: debug
  otlp/lightstep:
    endpoint: "ingest.lightstep.com:443"
    headers:
      "lightstep-access-token": "XXXXXXXX"
  otlphttp/splunk:
    traces_endpoint: "https://ingest.us1.signalfx.com/v2/trace/otlp"
    compression: gzip
    headers:
      "X-SF-Token": "XXXXXXXX"

#enables output for traces to xray
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [logging, otlp/lightstep, otlphttp/splunk]

but we are unable to inject the relevant secrets i.e. API Keys from secrets manager into this config.

While committing the configuration along with our lambda code works, this raises multiple issues:

1. Any secrets get committed to git
2. Updating api keys requires changing and redeploying all the lambdas

We would prefer if there was a mechanism to obtain this configuration file, or at least the access keys from secret manager in this layer automatically.

[iulspop]

@rijulg You can reference an environment variable in a collector configuration like this:

  otlp/lightstep:
     endpoint: "ingest.lightstep.com:443"
     headers:
       "lightstep-access-token": "${LIGHTSTEP_ACCESS_TOKEN}"

See OTel docs.

I just set the environment variable of the Lambda when I create it with the CDK (using env vars from my local dev environment). The Lambda env vars are encrypted at rest by default.

If you store the API keys in Secrets Manager, you can set retrieve them and set the Lambda env var when you deploy or update the env vars after you deploy with some script.

[iulspop]

The discussion in this issue references a number of proposals for external, dynamic configuration, especially for secrets.

[rijulg]

@iulspop the issue here is setting up the environment variables (or any other way of injecting the secrets) in the lambda before the Otel extensions boots up. We want to avoid hardcoding secrets in the code or deployment pipeline, and want to use SecretsManager to inject the secrets at runtime.

Since the Otel extension (and in fact every lambda extension) boots up in parallel there is no way to add custom code that can inject the secrets into the environment reliably before the Otel extension instantiates. The only solution so far available to us is building a custom layer that fetches the secrets and/or the entire config from another service like SecretsManager or ParameterStore before instantiating the Otel specific code.

[seh]

I just set the environment variable of the Lambda when I create it with the CDK (using env vars from my local dev environment). The Lambda env vars are encrypted at rest by default.

This approach still makes me uneasy, because while it's easy to understand and predict the operational outcome, it leaves these secret values visible to anyone inspecting the Lambda function's configuration. (That is, they can view the environment variable values, even if AWS encrypts them where it stores them.)

It would be nice if there was a way to tell this layer to fetch secrets from AWS Secrets Manager and bind them their values to environment variables before starting the OpenTelemetry Collector. Consider:

OPENTELEMETRY_COLLECTOR_ENV_SECRET_MAPPING=VENDOR_API_KEY:secret-name-1,OTHER_PASSWORD:secret-name-2

Setting that would cause the "VENDOR_API_KEY" environment variable to be bound to the value of the secret named "secret-name-1" and the "OTHER_PASSWORD" variable to be bound to the value of the secret named "secret-name-2".

With that, we could then use ${env:VENDOR_API_KEY} in our OpenTelemetry Collector configuration.