Add sub command to generate sample configuration for the collector

[neelayu]

trafficstars

Description: Adding a feature - Create a subcommand that generates a sample configuration for the collector. It has support for UI based options(like npm init), or via flags.

Link to tracking Issue: #5654

Testing: Unit Tests

Documentation: Need suggestions

Waiting for suggestions to improve the code more and opinions on how we can improve the documentation.

Sample workflow:

otelcorecol git:(config-sub) ✗ go run . config --help
Print sample configuration

Usage:
  otelcorecol config [flags]

Flags:
  -f, --config-file string      Name of the config file (default "otelcol.config.yaml")
  -d, --dry-run                 Dry run
  -e, --exporter stringArray    Exporters to be configured
  -x, --extension stringArray   Extensions to be configured
  -h, --help                    help for config
  -l, --pipeline stringArray    Specify the pipelines to be configured
  -p, --processor stringArray   Processors to be configured
  -r, --receiver stringArray    Receivers to be configured

If no flags are provided, use prompts It uses github.com/AlecAivazis/survey go library.

With flags

Output(only some parts for relevance)

exporters:
  logging:
    # LogLevel defines log level of the logging exporter; options are debug, info, warn, error.
    loglevel: 0
    # SamplingInitial defines how many samples are initially logged during each second.
    sampling_initial: 2
    # SamplingThereafter defines the sampling rate after the initial samples are logged.
    sampling_thereafter: 500
  otlp:
     ...

 ...
service:
  ..

As you can see, comments are also added for better understanding. This is achieved by github.com/open-telemetry/opentelemetry-collector-contrib/cmd/configschema

Suggestions are welcome to improve this PR. Thanks!

[codecov[bot]]

Codecov Report

Base: 91.76% // Head: 91.93% // Increases project coverage by +0.16% :tada:

Coverage data is based on head (5995f34) compared to base (e981169). Patch coverage: 94.84% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5943      +/-   ##
==========================================
+ Coverage   91.76%   91.93%   +0.16%     
==========================================
  Files         217      218       +1     
  Lines       13350    13770     +420     
==========================================
+ Hits        12251    12659     +408     
- Misses        870      875       +5     
- Partials      229      236       +7     

Impacted Files	Coverage Δ
service/configinit.go	94.11% <94.11%> (ø)
component/component.go	100.00% <100.00%> (ø)
component/exporter.go	100.00% <100.00%> (ø)
component/extension.go	100.00% <100.00%> (ø)
component/processor.go	100.00% <100.00%> (ø)
component/receiver.go	100.00% <100.00%> (ø)
exporter/loggingexporter/factory.go	100.00% <100.00%> (ø)
exporter/otlpexporter/factory.go	91.04% <100.00%> (+0.13%)	:arrow_up:
exporter/otlphttpexporter/factory.go	87.09% <100.00%> (+0.14%)	:arrow_up:
extension/ballastextension/factory.go	100.00% <100.00%> (ø)
... and 10 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

[svrnm]

Hey @neelayu, thanks for creating this. That's much more than I expected when I raised #5654!

I did not review the code, leaving that to the @open-telemetry/collector-approvers, but I tried out the functionality:

I like the prompts! Two issues:

If I skip all prompts I get the following YAML:

exporters: {}
processors: {}
receivers: {}
extensions: {}
service:
  # Specify extensions for the collector
  extensions: []
  pipelines: {}

Is it possible to not have {} for an empty set?

If I select all/some attributes it fails with

2022/08/23 12:03:16 open ../../go.mod: no such file or directory

This seems to be a hard-coded file path, when I copy my compiled collector binary into a folder like "./bin/bin" within the forked repository things work. You may need to verify that the final collector works independent of the existence of these files.

To mention that: I like that if I only provide the -f <filename> I get an empty file and not the "missing" prompts. This is really helpful if I just need an empty config that I might like to edit in a second step.

Regarding the copied comments: On the one hand I like them, on the other hand they can make a config extremely hard to read for a new-comer, e.g. otelcol config -p batch -e otlp -d creates a config with 106 lines. Maybe you can provide another flag that enables/disables to fill out the config with all the attributes + comments?

[neelayu]

@svrnm Thanks for going through this PR and giving it a try.

If I skip all prompts I get the following YAML: yaml code Is it possible to not have {} for an empty set?

Yes. This is something which I want an opinion on. I'd like to add validations to check if the inputs were provided or not.

If I select all/some attributes it fails with

2022/08/23 12:03:16 open ../../go.mod: no such file or directory Correct. I have to fix this!

on the other hand they can make a config extremely hard to read for a new-comer, They do make the configuration file pretty large. As you mentioned, are these options better?

otelcol config -p batch -e otlp --no-comments

Will produce all fields with no comments.

otelcol config -p batch -e otlp --no-attrs

Will produce only the configuration file with key headers?

exporters:
  otlp:
     # For configuration params, refer https://pkg.go.dev/go.opentelemetry.io/collector/exporter/otlpexporter

As you can see, there are no attributes, but a single line comment indicating where the info on the configuration can be found. I believe pkg.go.dev will do the job.

I think its difficult to get only "mandatory" or "minimal" configuration fields from the struct. So its either all or none.

[svrnm]

@svrnm Thanks for going through this PR and giving it a try.

If I skip all prompts I get the following YAML: yaml code Is it possible to not have {} for an empty set?

Yes. This is something which I want an opinion on. I'd like to add validations to check if the inputs were provided or not.

I personally think it's ok to leave them empty.

on the other hand they can make a config extremely hard to read for a new-comer, They do make the configuration file pretty large. As you mentioned, are these options better?

otelcol config -p batch -e otlp --no-comments

Will produce all fields with no comments.

otelcol config -p batch -e otlp --no-attrs

Will produce only the configuration file with key headers?

exporters:
  otlp:
     # For configuration params, refer https://pkg.go.dev/go.opentelemetry.io/collector/exporter/otlpexporter

I like that, teaches people also where they can look for details!

I am only wondering what should be the default (full details, only attributes, no comments&no attributes)?

As you can see, there are no attributes, but a single line comment indicating where the info on the configuration can be found. I believe pkg.go.dev will do the job.

I think its difficult to get only "mandatory" or "minimal" configuration fields from the struct. So its either all or none.

That's an interesting one! What about something like:

otelcol config -p batch -e otlp,endpoint=https://mybackend/,entpoint.headers.apiKey=asdf,tls.insecure=true --no-attrs

gives you some of those attributes set (and skips the rest)?

[neelayu]

Hey @svrnm I have made a few changes. I didnt really get time this week to work on it. Re:

That's an interesting one! What about something like:

otelcol config -p batch -e otlp,endpoint=https://mybackend/,entpoint.headers.apiKey=asdf,tls.insecure=true --no-attrs

gives you some of those attributes set (and skips the rest)?

I believe this is as good as knowing what params are present. So the user might as well use that and create a config file.

[svrnm]

@neelayu thank you, I am looking forward to have this feature. I am eventually just a consumer of that, as an end-user but also to simplify the Getting Started docs for the collector over at opentelemetry.io :)

[linux-foundation-easycla[bot]]

The committers listed above are authorized under a signed CLA.

• :white_check_mark: login: neelayu / name: Neelay Upadhyaya (aecaaf22c7c07a41ce0caa8a9da5756b4db7f71f, b9323b4d24388150d147b2775fe821be4e2f2a7c, caf719fafad3fcfb9c3cb766089036e16d911b88, b6b41af3c55f22281eca57ce8aed4d472e595eb7, 044642b3e1114ed3e32611bc89c65673778ecea7, fbd463cbfe206a4c4ee86338de596d62fdb5ac29, 2a1b2b29fc2080626fcf2c7d48648d5a4aee57a2, 1244b3353771d5f85f7414a5c586d0d8dcd27d4a, e0186cc70045f0d43b60626a08259f8e22c2c4ed)

[neelayu]

I have refactored the code. The basefactory essentially adds a SampleConfig() method. The authors are expected to return a configuration for the component. This is more or less what Telegraf agent does in its approach. Since I didnt want to introduce breaking changes across the contrib repo as well, I have added it as an option. Let me know if this approach works. @jpkrohling

[github-actions[bot]]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[svrnm]

I am still a big fan of this and keen to see this happening, is there anything I can do to help this progressing? ty

[mx-psi]

@jpkrohling is there anything needed here?

[neelayu]

@Aneurysm9 I did not add *sample.yaml in my first few drafts. You can check the approach which I used here https://github.com/open-telemetry/opentelemetry-collector/commit/044642b3e1114ed3e32611bc89c65673778ecea7

In that I used the configschema API from contrib repo to fetch the fields and doc info. The limitation of that approach is that we can't obtain sensible defaults for fields which are mandatory.(Eg endpoint)

[neelayu]

Thanks @jpkrohling for the comments. I just want to ensure that this approach of sample config yaml makes sense. One other approach I can think of creating a web equivalent config creation which was discussed in SIG meeting in CLI. That is for every field we ask the user to input the value, the docstring will act as help text. Along with it, we can have an option of --full which will create the configuration with no user input, the one which I created in my first commit. Let me know which approach we want to take.

If we stick to sample yaml approach, I am aware that the yaml files will have to be created via make targets. However, that would add too much code to review in this PR. So I can take it up in a separate task. Although there is one caveat here, the files will be auto-generated, but they would be bulky with many fields and large comment sections. So the onus lies on the developer(author of the component) to select(or not) which ones are relevant for a sample config.

[github-actions[bot]]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[jpkrohling]

Sorry, I need to catch-up with the collector's current state before giving any meaningful input here. Perhaps @codeboten knows what's the state around this part, especially regarding the UI that was shown some weeks ago to help with the collector's config?

[github-actions[bot]]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[github-actions[bot]]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[github-actions[bot]]

Closed as inactive. Feel free to reopen if this PR is still being worked on.