fluent
Feature request: multiple labels per source
It would be better to be able to route logs from one source to multiple labels, so that it will be much easier to use logs for different purposes, such as storing and monitoring.
Indeed it can be configured using copy and relabel, but such a combination tends to be indirect and messy, especially when dealing with a lot of sources.
Example: save all log records and monitor errors in them
With multiple labels in one source, it can be written as below.
<source>
@type tail
@label @cloudlogging @app_monitor
tag app
path /path/to/app.log
format /(?<severity>.+?) +(?<message>.*)/
</source>
<label @cloudlogging>
<match **>
type google_cloud
</match>
</label>
<label @app_monitor>
<filter **>
@type grep
regexp1 severity ERROR
</filter>
<match **>
type sensu
check_status CRITICAL
</match>
</label>
Currently, we must use copy and relabel instead.
<source>
@type tail
@label @app_raw
...
</source>
<label @app_raw>
@type copy
<store>
<match **>
@type relabel
@label @app_monitor
</match>
</store>
<store>
<match **>
@type relabel
@label @cloudlogging
</match>
</store>
</label>
...
For supporting multi-labels, I'm considering error handing. This is why fluentd doesn't have copy like feature in the core. https://github.com/sonots/fluent-plugin-copy_ex
Do you have an idea for error cases?
I understood the change is not trivial. But it still seems good to have a standard way for error handling in such situation.
I think the following is an adequate policy.
- Rescuing errors unhandled in each output labels,
- making a composite error instance (such as
CompositeError), - and raise it to the input plugin.
IMO, having 2 or more labels in @label is too complex from viewpoint of whole architecture.
How about to merge relabel feature to copy and provide configuration syntax like this?
It's much easier to implement, and behavior for errors are also understandable for users.
<source>
@type tail
@label @app_raw
...
</source>
<label @app_raw>
<match **>
@type copy
<relabel>
@label @app_monitor
</relabel>
<relabel>
@label @cloudlogging
</relabel>
</match>
</label>
Do you mean followings will have different behaviors in terms of error handling? It looks complicated for users to master usages of fluentd. We can not guess different points from their configuration words.
<label @app_raw>
<match **>
@type copy
<relabel>
@label @app_monitor
</relabel>
<relabel>
@label @cloudlogging
</relabel>
</match>
</label>
and
<label @app_raw>
<match **>
@type copy
<store>
@type relabel
@label @app_monitor
</store>
<store>
@type relabel
@label @cloudlogging
</store>
</match>
</label>
