graylog2-server
graylog2-server copied to clipboard
Graylog2
"Default" configuration for Sidecar Log Collector
When Sidecar is installed on 1000s of machines, it is unpractical to assign configurations manually. Therefore, there should be a way in the UI to set a default configuration for each Log Collector type.
I fully agree that assigning configurations manually is unpractical and a pain if you have to do it frequently. I also agree that there should be a more effective way to establish the assignation. The tags used in previous version did that role.
The current plan is as follows:
We will "re-introduce" tags into the sidecar.yml
This allows users to decide whether they want to assign configurations manually, or automatically via tags.
The Sidecars that have no tags will behave like before: A configuration needs to be assigned manually. If a Sidecar reports to Graylog that it has a set of tags, the server will assign it the configurations that match the tags. A Sidecar with tags will not be allowed to have its configuration assignment changed manually via the UI. This should be visually indicated somehow.
Open questions:
- What happens if multiple configurations of the same collector match a tag? We can only assign one. Which configuration wins?
Potential pitfalls:
- The config lookup via tags needs to be made efficient
- What happens if multiple configurations of the same collector match a tag? We can only assign one. Which configuration wins?
@mpfz0r I think we cannot decide that automatically. How about starting one collector instance for each configuration? The old sidecar system rendered a combined collector configuration on the sidecar node to avoid creating multiple instances. But since the new sidecar system works differently, that's not an option.
Combining configurations will only work if we introduce another type of configuration. The current configurations are "full" config files. If we had a "snippet" type, we could probably combine the configurations to allow starting a single instance. That also requires more knowledge about the collectors in the server backend, of course. Right now, the collectors are just binaries, command-line options, and config templates. If we want to combine configuration files, we would need to introduce collector types like "filebeat" that have a piece of backend code that knows how to combine configs for filebeat.
So, starting separate instances sounds more straightforward than the other option. :smile: It still requires changes like ensuring unique data and log directories for each collector+config instance.
Is there another option I have missed?
A Sidecar with tags will not be allowed to have its configuration assignment changed manually via the UI. This should be visually indicated somehow.
@mpfz0r Can you elaborate on why manually assigning configurations to a sidecar with tags should not be possible?
For testing purposes, it would be nice also to assign configurations manually. Otherwise, the sidecar config needs to be modified to add "testing" tags.
In the end, assigning configurations based on tags or manually is similar. A rule system could implement both. It selects configuration assignments for nodes based on node facts.
Assign based on tags: assign config(a) if node.tags.include(yolo)
Assign based on node ID: assign config(a) if node.id == abc-123
Is there another option I have missed?
No what you described sounds like a possible solution. A simpler one would be some kind of restriction that prevents users from creating these conflicting tagged configurations in the first place
Is there another option I have missed?
No what you described sounds like a possible solution. A simpler one would be some kind of restriction that prevents users from creating these conflicting tagged configurations in the first place
@mpfz0r Aye. Then you are back to duplicating configurations for the different combinations you need, though. (if you can only have a single filebeat config per sidecar, for example)
@mpfz0r Can you elaborate on why manually assigning configurations to a sidecar with tags should not be possible?
I wanted to keep this as simple as possible. (This is a misc topic right now) Providing both opens up a lot of edge cases we would need to consider and catch. E.g. the UI would need to treat manually and automatically assigned configurations differently. If we make this decision on each sidecar it becomes much simpler.
We've had a long conversation about what needs to be done, what UI changes would make sense, and the overall behavior, and I believe this is way too large for a few days of misc work. At the very least we need to describe the behavior better and should then pitch the work for the next cycle. Even if the ideas presented here make sense, the amount of work is just too much.
Hi everyone, we understand that managing a larger fleet of Sidecars using different configurations is currently quite cumbersome. The team is now working on a related improvement, which we can hopefully share soon.
