fluent-plugin-webhdfs icon indicating copy to clipboard operation
fluent-plugin-webhdfs copied to clipboard

Published 20 hours ago • verified publisher icon fluent

根据日志中的字段指定hdfs上的路径

Open zhangzhengstrive opened this issue 8 years ago • 5 comments

您好，请问怎么根据日志中的某个字段指定hdfs上的路径path

zhangzhengstrive avatar Nov 22 '16 14:11 zhangzhengstrive

Write an issue in english.

repeatedly avatar Nov 22 '16 14:11 repeatedly

Hello, How to use the log field to specify the hdfs path

zhangzhengstrive avatar Nov 23 '16 07:11 zhangzhengstrive

It's impossible right now, but will be supported in future version with Fluentd v0.14.

tagomoris avatar Nov 23 '16 11:11 tagomoris

you may use rewrite_tag_filter to rewrite the tag, and load hdfs path from tag

maxwellzdm avatar Nov 24 '16 03:11 maxwellzdm

thank you, let me try. How analytical chunk? how string into chunk? chunk.msgpack_each do |tag, time, record| my_domain = record['domain'] end def format(tag, time, record) [tag, time, record].to_msgpack end But the data stored in hdfs is garbled compress_context(chunk) do |data| send_data(hdfs_path, data) end

zhangzhengstrive avatar Nov 25 '16 01:11 zhangzhengstrive