fluent-plugin-webhdfs

fluent-plugin-webhdfs copied to clipboard

kerberos_keytab not authorize

2

Hi. I try to connect to kerberized hdfs cluster. But can't. Set "kerberos_keytab" to config file - but nothing change. Only if i mannually kinit user by keytab it works....

Vhodnoylogin

Hi All, Please help to solve this issue. we are using fluentd as our central log forwarder. we are getting lot of backlog records in file buffer due to webhdfs...

lakhsaaron

systemctl reload td-agent.service is not working properly with this type under type copy. restarting the service or stop/start make the configuration work normally and transfer data to hdfs.

mojili

``` @type webhdfs path /tmp/fluentd/test/test.log host myhttpfs.example.com port 14000 httpfs true username admin flush_interval 5s ``` ``` 2020-01-30 19:17:31 +0900 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2020-01-30 19:17:32...

hahafamilia

httpFS - Do not create file if it does not exist

5

Hello, We are running a MapR custer and webHDFS is not supported by MapR. So we are trying to populate hadoop using httpFS. Our Webhdfs config : ``` @type webhdfs...

hurtauda

use chunk_id in path

3

According to your documentation it is allowed to use path "/log/access/%Y%m%d/#{Socket.gethostname}.${chunk_id}.log" In my case path "/raw/syslog/logdate=%Y%m%d/loghour=%H/#{Socket.gethostname}.${chunk_id}.journald.log" failed to communicate hdfs cluster, path: /raw/syslog/logdate=20190625/loghour=07/fluentd3.2019-06-25 08:11:10 +0000 [warn]: #10 ${chunk_id} is not...

bobykus31

Here is my config ``` @type copy deep_copy true @type webhdfs namenode master1.hdoo.com:50070 standby_namenode master2.hdoo.com:50070 path "/raw/iostats/%Y%m%d/%H/#{Socket.gethostname}.log" compress gzip # or 'bzip2', 'snappy', 'lzo_command' retry_known_errors yes retry_interval 65 open_timeout 180...

bobykus31

HA configuration performs incorrectly

4

Problem: I am running HDFS within my Mesos cluster. It is fully HA. I have configured a matcher to point to both NameNodes. However, when the first listed NameNode is...

wolf31o2

Can hdfs path use ${tag} or ${record} ? just like: /data/hdfs/${tag}.#{Socket.gethostname}.log

1

Hello, I use 'fluent-plugin-webhdfs' version '1.2.2', fluentd version: 1.4 My log file looks like: `interface=login&time=20170925&...others fields...` In my td-agent config file, I named my log as a tag:interface.login.hdfs ``` @type...

monkeypiggy

根据日志中的字段指定hdfs上的路径

5

您好，请问怎么根据日志中的某个字段指定hdfs上的路径path

zhangzhengstrive