Print configuration used by OTEL Collector at runtime

[Mario-Hofstaetter]

trafficstars

Is your feature request related to a problem? Please describe.

otelcol can be configured by one or multiple yaml configuration files, which also support environment variable expension. Therefore, especially when using ENV vars, it is useful to be able to view the config otelcol is actually running with.

AFAIK currently there is no such feature.

Below I will suggest solutions I am using on a daily basis with other observability tools.

To cite @mx-psi from slack:

I think there are some security concerns here that would need to be addressed (re: some components need tokens/keys in the config that we wouldn’t want to show), but it definitely sounds like something useful

Somewhat remotely related to that concern: #2466

Describe the solution you'd like

All solutions should consider secrets like api tokens, any maybe censor them ******

Print configuration in command line

Probably the easiest way. For example Grafana promtail has a CLI flag -print-config-stderr which just dumps the config into the commandline, including expanded ENV vars. If the seperate -config.expand-env flag is omitted, it is clearly visible where ENV vars are used in the configuration. This could be done for otelcol in exactly the same way (however ENV vars are expanded by default in otelcol)

./promtail -print-config-stderr "-config.file=promtail-config.yaml" "-config.expand-env"

Regarding secrets, I did a test using promtail basic_auth, and the password is actually masked with <secret> in the console output:

clients:
- url: https://ax-monitor-02.ax.tech/loki/loki/api/v1/push
  batchwait: 1s
  batchsize: 1048576
  basic_auth:
    username: foobar
    password: <secret>
  tls_config:
    insecure_skip_verify: true
  follow_redirects: false

They also support a -dry-run flag, so nothing is actually pushed (otelcol equivalent for dry-running would be: Don't push logs & traces, don't expose metrics), but I guess that could be a feature request on its own.

Show configuration in HTTP endpoint

Otelcol has several HTTP endpoints (none active by default?), which could enable showing the active configuration in plain text. For example, using extensions: [health_check , zpages] adds two seperate http endpoints.

Maybe an optional extension "config" (lack of a better name) could be configured in a similar fashion to expose the configuration via http.

• When running prometheus, its config (html) is available in http://localhost:9090/config

Click to show prometheus web config

Regarding secrets, prometheus does seem to hide secrets in the config:

• When running promtail, the config (html) is available in its http server: http://localhost:9080/config

Click to show promtail web config

Regarding secrets, I did a test using promtail basic_auth, and the password is actually masked with <secret> in the web endpoint too:

• When running Grafana Loki, the server exposes it's config as plaintext on http://localhost:3100/config

You get the idea.

Describe alternatives you've considered

?

[mx-psi]

All solutions should consider secrets like api tokens, any maybe censor them ******

To expand on the problem here: it's hard to know what options are an API token, and we probably don't want to have an opt-in system for this where people declare "this field is an API token", since the probability that we or downstream distros forget to mark something as an API token is too high, and it conflicts with the "secure by default" philosophy of the OpenTelemetry project.

As a start, we can assume some types are safe to show by default (we can always show bools, ints, time.Durations...) and other types are unsafe to show by default (basically anything with strings in it) and provide a way to explicitly state if a field is safe to show. A simple approach is to mark fields with a sensitive="true/false" struct tag, but still there are edge cases to consider (what about HTTP headers configuration in the OTLP HTTP exporter? Some vendors use them to send API tokens, so the values need to be censored, but the keys can be shown, and it's not easy to express this using a struct tag).

I think this is the first problem that needs to be solved for this feature to eventually happen.

[Mario-Hofstaetter]

@mx-psi I updated my original post with examples who secrets are handled by prometheus and promtail. It looks obvious promtail is sharing the code with prometheus in that manner.

I have no idea how they implemented it interally, but as per documentation: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file https://grafana.com/docs/loki/latest/clients/promtail/configuration/#generic-placeholders

https://github.com/prometheus/prometheus/blob/685ce9964da2bcfdfa72a89db48d889a9457574e/docs/configuration/https.md?plain=1#L25

<secret>: a regular string that is a secret, such as a password

So maybe OTEL can look into how the folks of prometheus have done it.

where people declare "this field is an API token", since the probability that we or downstream distros forget to mark something as an API token is too high

But I guess this will always be a possiblity that is hard or impossible to protect against?

[bogdandrutu]

@Mario-Hofstaetter @mx-psi first of all, thank you for this issue.

I have some confusions/questions here:

• I am confused about the meaning of "--dry-run", we can discuss if "-dry-run" means actually running and printing to console, or just print the config.
• mx-psi, curious about your concern related to security here. If someone has the configuration and privileges to run the collector that has access to the "token" or anything "secure" why do you see a problem with us printing that to console when a specific flag/command is given? But if we expose this on an endpoint (since we don't have any ACL implemented), then not sure what we can do there, probably a tag is safe to do in my opinion and use some kind of "" string.

PS: @Mario-Hofstaetter metrics are also "pushed" and not exposed :) This is a bit of a biased coming from prometheus world.

[Mario-Hofstaetter]

@bogdandrutu

we can discuss if "-dry-run" means actually running and printing to console, or just print the config.

Neither, I guess I should not have mixed it in here. The two flags can be combined, but seperately they mean:

• (in promtail), -print-config-stderr dumps the config to console and then starts its normal operation. It is not possible to just dump the config & then exit, which I personally would prefer.

To keep this issue compact, I could move the dry run feature to another issue if you'd like? It is a different feature.

• (in promtail) -dry-run does not print the configuration, but instead prints information about what the program is doing to console, and not writing any data to files or endpoints. I guess the OTEL collector äquivalent to this behavior is using the logging exporter ?

• promtail has yet another flag --inspect which prints debug info for its pipelines, I guess this is equivalent to using

exporters:
  logging:
    loglevel: debug

to print infos about receiver, processor, export behavior.

For OTEL collector, dry-running would IMHO mean:

• Do not export any data to endpoints (no traces, logs, metrics),
• however otelcol by its architecture does not know what its exporters are doing? The logging exporter mentioned above would be used for diagnosis during dry-running, and should therefore not be disabled by dry-running

However I am not entirely sure what is logged by telemetry logs in debug level and what by using the logging exporter in a service pipeline in debug level

[mx-psi]

mx-psi, curious about your concern related to security here. If someone has the configuration and privileges to run the collector that has access to the "token" or anything "secure" why do you see a problem with us printing that to console when a specific flag/command is given?

@bogdandrutu I was thinking about a similar feature available in the Datadog Agent, where the API/application keys are censored, but I didn't have a security model in mind, so it may be possible that we can get away without doing this if the approach is different. IIRC in the Datadog Agent we do this by exposing it on an endpoint, and the output is also something that gets sent to Datadog for technical support, so I assume these are the main reasons why we want edto remove the keys from the output in the Datadog case.

But if we expose this on an endpoint (since we don't have any ACL implemented), then not sure what we can do there, probably a tag is safe to do in my opinion and use some kind of "" string.

I guess if we support changing the configuration at runtime, we want to expose it in an endpoint, right? Also, with 'a tag', do you mean an opt-in or opt-out tag? And lastly, do you have thoughts about edge cases like how to censor HTTP headers on, say, the OTLP HTTP exporter?

[ringerc]

For starters a --dump-config option to the validate subcommand, or a dump-config subcommand would help.

Sure it doesn't let you get the config from the running collector. But at least it provides a way to see the merged config after file inclusions and processing of multiple input files.

Right now I'm struggling with an issue where validate is quite happy with my config but at runtime it fails due to a reported issue with a connector being used as a receiver but not an exporter. Except... it is used as an exporter, in a different config that's overlaid. I have no way to see the final config that the collector produces from the inputs in order to try to understand how it's merging them and how it might be misunderstanding the intent of the config I wrote. Because the config issue causes the collector to bail out, querying runtime config wouldn't work anyway. But validate doesn't complain about it...

[mx-psi]

@ringerc Adding functionality to the validate subcommand seems independent from this issue (solving this issue would not help with your problem since you don't have a running Collector). Could you file a separate issue for the 'validate' ask?

[ringerc]

@mx-psi The validate issue was handled separately. I was just using it as an example for why it's important to be able to see the config as actually resolved after merging of sections etc.

[yurishkuro]

Suggestion: --print-config-insecure, which prints everything without masking. Per Bogdan's comment, the user running such command already has runtime access to all secrets. For people who want an output with masking, let's keep another issue if someone wants to come up with a solution. But right now that extra requirement is blocking simple implementation of a much needed feature.

[chenlujjj]

Hi, any updated on this issue? This is a very helpful feature, especially when we have multiple source of configs

[mx-psi]

The OpAMP folks have been making some progress on this, see #6833 and more recently #10282. I think the work OpAMP is doing could be leveraged here if someone is willing to pick this up.

cc @evan-bradley, do you have any thoughts on this?

[evan-bradley]

This will be possible with sensitive values redacted once https://github.com/open-telemetry/opentelemetry-collector/pull/10139 is complete. I plan to work on displaying this in the zPages extension afterward.

Would we like a command/flag to display this information as well, or would the zPages extension be sufficient?

[mx-psi]

Would we like a command/flag to display this information as well, or would the zPages extension be sufficient?

zPages/something similar would be required to fully support this if configuration changes at runtime. I am not opposed to a command/flag as well if users ask for it

[chenlujjj]

+1 for the command/flag.

Here is our use case: we are using the splunk-otel-collector, and plan to use the include config source feature which combines multiple template or plain files into one config file. Printing configuration via command/flag can surely help users understand what the complete config looks like before running the collector.

[ringerc]

I landed up dropping all use of DataDog so as the original requestor I won't be able to contribute here.

[evantorrie]

+1 for command flag. Ideally this should work even if the configuration is not able to instantiate a fully operating collector -- e.g. if there were some error that only manifested itself during component startup.

The reason is that this would allow us to inspect the resolved configuration (again, similar use case to that mentioned above where we are either including additional configs from remote sources that get merged in to the final config) prior to collector startup.

[yurishkuro]

@evantorrie that's a great point. I see two different modes for this, one is "show exactly what I configured", which takes care of any resolutions and/or config merging, but before any of the collector logic kicks in. This can never fail. And the 2nd mode is to print the final configuration after parsing and defaults are applied - this might fail due to unmarshaling errors.

[alexylew]

Is anyone actively working on this? We're also making heavy use of providing multiple config files to a collector and need this feature for debugging. @evan-bradley anything in flight? I'd love to contribute.

[yurishkuro]

This was the recent work I've seen https://github.com/open-telemetry/opentelemetry-collector/pull/10694

[mwear]

Similar functionality is available in healthcheckv2extension, which is currently experimental. See: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/healthcheckv2extension#collector-config-endpoint. Work needs to be done to omit sensitive data though.

[VihasMakwana]

+1 for command line flag.

I propose following:

• Enhance the validate subcommand with the --dump-config option, which will display the configuration after it has been resolved/expanded/merged from different sources.
• Add another option as a follow-up PR, --dump-mode, which will show both the configuration after resolving/expanding and the configuration after applying default settings.

Usage example(s):

1. To print config after resolving and expanding which shoes config provided by user.

otelcol --config one.yaml --config two.yaml --validate --dump-config

2. To print config after resolving, expanding and applying defaults.

otelcol --config one.yaml --config two.yaml --validate --dump-config --dump-mode full

---

@mx-psi @bogdandrutu Let me know your thoughts over this and if this sounds good by you, can you assign this to me and I'll work on rolling out a PR soon.

[mx-psi]

I support doing this. Maybe we can split into two issue for the two use cases we have seen here so far? One issue for a new subcommand/CLI flag option, another issue for a new component that exposes an HTTP endpoint. I can assign you to the individual issues if created, @VihasMakwana :)

Enhance the validate subcommand with the --dump-config option, which will display the configuration after it has been resolved/expanded/merged from different sources.

People may want to check the configuration even if invalid, I feel like this should be separate from validate (maybe a separate subcommand?)

Add another option as a follow-up PR, --dump-mode, which will show both the configuration after resolving/expanding and the configuration after applying default settings.

I feel strongly that this should be done through an extension that exposes and HTTP endpoint, to support config updates at runtime. I am happy to sponsor such an extension if proposed. This extension can live in opentelemetry-collector-contrib.

[VihasMakwana]

People may want to check the configuration even if invalid, I feel like this should be separate from validate (maybe a separate subcommand?)

I agree. This can be a separate subcommand. Here's an issue for it. @mx-psi

I feel strongly that this should be done through an extension that exposes and HTTP endpoint, to support config updates at runtime. I am happy to sponsor such an extension if proposed. This extension can live in opentelemetry-collector-contrib.

Makes sense. I'll work on proposing a new extension for this and probably create a new issue on contrib repo.

[khushijain21]

I feel strongly that this should be done through an extension that exposes and HTTP endpoint, to support config updates at runtime. I am happy to sponsor such an extension if proposed. This extension can live in opentelemetry-collector-contrib.

Currently, the healthCheckExtension already exposes the config on an HTTP endpoint. Having another extension with the same feature would be an overdo.

Support on config updates will require changes on collector core to receive updates from the extension. Check this doc https://github.com/open-telemetry/opentelemetry-collector/blob/ff09ab196c59c1049469419d00bd37ca132e5580/docs/service-extensions.md#configuration-and-interface

which mentions

There are more complex scenarios in which there can be notifications of state changes from the extensions to their host. These more complex cases are not supported at this moment, but this design doesn’t prevent such extensions in the future

It may be a better idea to have support for config reload directly on the collector. wdyt? Issue to follow on collector for config reloading - https://github.com/open-telemetry/opentelemetry-collector/issues/5966

[VihasMakwana]

For starters, I think it makes sense to enable providers to send events to the watcher. But in my opinion, this behaviour should be disabled by default and can be enabled via feature flag.

We can change fileprovider as a first step.

Wdyt @mx-psi ?

[mx-psi]

Currently, the healthCheckExtension already exposes the config on an HTTP endpoint. Having another extension with the same feature would be an overdo.

Right, I wasn't aware v2 was adding this. No need for a separate extension then.

It may be a better idea to have support for config reload directly on the collector. wdyt? Issue to follow on collector for config reloading - https://github.com/open-telemetry/opentelemetry-collector/issues/5966

I don't understand this, I am not saying the extension should be the one handling the configuration reload. I am saying a subcommand doesn't work if there are configuration reloads.