Configuring logger, metrics, tracers, etc inside my own GoLang distribution

[lucasoares]

Hello.

I have a few custom components for benthos implemented in my own repository and I have several teams using this custom runnable. Currently each team can configure logger, traces, metrics, etc inside their config files.

The problem is that we are implementing few rules and formats that every team must follow but with the current state of benthos we would need to copy and paste this configuration for every single config file of our teams, and no one guarantees that new files will have this configuration

The best scenario for me and my team would be benthos allowing a injection of how the logger is made (being able to inject a interface implementation to override benthos internal logging, for example). But for now the configuration directly on my own golang files would be great!

I'm also happy to contribute, just need some guidance hahaha

[mihaitodor]

Hey @lucasoares, I'm wondering if you considered using CUE instead of yaml for managing configs. With CUE, you could develop a library of standard components which can be used by everyone. If not, I guess it might be handy to allow users to either pass the -c argument multiple times or maybe support wildcards for -c. WDYT?

For injecting the logger for custom plugins, there's also #1155 and #1080. Please feel free to chime in there too or upvote them.

[lucasoares]

Unfortunately CUE is not an options for my company... About passing -c argument multiple times it would help for now, since I would be able to force the same config.yaml for logging, metrics, etc..

The best solution tho would be a public API allowing users to inject the logging provider or something like this...

For now I will create my own logger component but I may have to fork benthos and change the logger implementation to follow our standards..

[mihaitodor]

OK, I'll leave this issue open to consider expanding the -c parameter. Forking Benthos is definitely not ideal, but I'm not sure when the other 2 issues will be addressed...

[lucasoares]

@mihaitodor in the case of expanding the -c parameter. The result config file would be a deep-merge of all config files? Or we should make something simple by just appending the files?

I want to make a PR with this because is a must-have for my company.. For now I'm manually appending the file with the configurations we need to set for all pipelines, having this native would be nice..

[lucasoares]

@mihaitodor this commit expands the -c parameter to receive multiple config files: https://github.com/lucasoares/benthos/commit/560678c76f13e07422833893cbaab2e7158e28df

In this scenario I'm concatenating all files into a single file.

The problem in this approach is because lint will only run for the result of the concatenation (making it difficult to know which file and which line the error is).

A better approach maybe is to run the lint in each file separately and then merge the files. But I would need to run the lint in the final file to guarantee that is a valid configuration.

What do you think? Is this something worthy to have inside benthos codebase? For my company this would be really great.

I also could make a deep merge between files but I think the solution would become very complex and I don't think this is something we want in benthos codebase

[mihaitodor]

Thanks for digging into that! I probably didn't think this through well enough, especially since it will mess up linting and it will get very confusing if any items need to be merged... Probably best to discuss with @Jeffail about this.

[mihaitodor]

Another idea I just had is that you might wanna' consider introducing a tool like Kustomize, where you add an intermediary step to aggregate individual configs into a final Benthos config. I think Kustomize doesn't work just for Kubernetes configs. Another one I'm aware of is gomplate, but it might be less convenient in this particular case.

[lucasoares]

Another idea I just had is that you might wanna' consider introducing a tool like Kustomize, where you add an intermediary step to aggregate individual configs into a final Benthos config. I think Kustomize doesn't work just for Kubernetes configs. Another one I'm aware of is gomplate, but it might be less convenient in this particular case.

Unfortunately, this would not work for my company. We have a homemade configuration management and deployment system that uses Consul to store configuration and access it on pod startup... Currently, we implemented this access to rewrite the config file and add our custom configs for logger, metrics, etc (which do almost the same thing that I did in the commit I provided) and it is working, the problem is that I can't emulate the same configuration on my CI and local environment easily, to test etc.

Thank you anyway!