fluent-plugin-webhdfs
fluent-plugin-webhdfs copied to clipboard
Can hdfs path use ${tag} or ${record} ? just like: /data/hdfs/${tag}.#{Socket.gethostname}.log
Hello,
I use 'fluent-plugin-webhdfs' version '1.2.2', fluentd version: 1.4
My log file looks like:
interface=login&time=20170925&...others fields...
In my td-agent config file, I named my log as a tag:interface.login.hdfs
<source>
@type tail
format none
path /data/td-agent/logs/interface.log
pos_file /etc/td-agent/pos/interface.pos
read_from_head true
tag interface.**login**.hdfs
</source>
<match interface.**>
@type webhdfs
host *.*.*.*
port 50070
path /data/gamelogs/hdfs/date=${record['time']}/${tag[2]}.#{Socket.gethostname}.log
<format>
@type json
</format>
</match>
and I need to get "login" from tag, and get time from my log record, so if there has any method to get these two values?
In some other plugins, I can use $tag[2] or ${record['interface']}, ${record['time']} to get these values,but in this plugin, it doesn't work. Thanks...
@monkeypiggy hello, I met this problem, too.
I solved it add some configuration.. like below
<match **>
@type webhdfs
----
path "/data/${tag}/%Y%m%d/%H/#{Socket.gethostname}.${chunk_id}.log"
<buffer tag,time>
</buffer>
</match>