fluent-plugin-webhdfs icon indicating copy to clipboard operation
fluent-plugin-webhdfs copied to clipboard

Can hdfs path use ${tag} or ${record} ? just like: /data/hdfs/${tag}.#{Socket.gethostname}.log

Open monkeypiggy opened this issue 7 years ago • 1 comments

Hello, I use 'fluent-plugin-webhdfs' version '1.2.2', fluentd version: 1.4 My log file looks like: interface=login&time=20170925&...others fields... In my td-agent config file, I named my log as a tag:interface.login.hdfs

<source>
  @type tail
  format none
  path /data/td-agent/logs/interface.log
  pos_file /etc/td-agent/pos/interface.pos
  read_from_head true
  tag interface.**login**.hdfs
</source>
<match interface.**>
  @type webhdfs
  host *.*.*.*
  port 50070
  path /data/gamelogs/hdfs/date=${record['time']}/${tag[2]}.#{Socket.gethostname}.log
  <format>
    @type json
  </format>
</match>

and I need to get "login" from tag, and get time from my log record, so if there has any method to get these two values?

In some other plugins, I can use $tag[2] or ${record['interface']}, ${record['time']} to get these values,but in this plugin, it doesn't work. Thanks...

monkeypiggy avatar Sep 25 '17 15:09 monkeypiggy

@monkeypiggy hello, I met this problem, too.

I solved it add some configuration.. like below

<match **>
  @type webhdfs
  ----
  path   "/data/${tag}/%Y%m%d/%H/#{Socket.gethostname}.${chunk_id}.log"
  <buffer tag,time>
  </buffer>
</match>

JiHyunSong avatar Mar 15 '18 06:03 JiHyunSong