Kubernetes plugin neither annotating nor logging errors

Open awcheney opened this issue 4 years ago • 1 comments

Hi there folks,

Not certain if this is a fluentbit-operator problem or a general fluentbit kubernetes filter plugin issue...

I've implemented the operator using helm into a namespace called 'logging', using images kubesphere:operator:fluentbit-operator:v0.12.0 and kubesphere/fluent-bit:v1.8.3; it's using the standard input (docker, kubelet and tail) and filter (containerd, kubernetes and systemd) resources along with the fluentd output resource (as defined by the helm chart) to forward logs to a fluentd statefulset aggregator where, for debugging, I'm currently using only the in_forward and out_file plugins.

The container logs in the 'logging' namespace are being annotated correctly - I'm seeing logs from other containers (I'm running a log-generating pod in another namespace called 'flogger').

Logs relating to the kubernetes filter from the fluent-bit pods look like this (here's one, the other is identical):  kubectl logs fluent-bit-k76wd | grep filter [2021/10/02 03:36:10] [ info] [filter:kubernetes:kubernetes.2] https=1 host=kubernetes.default.svc port=443 [2021/10/02 03:36:10] [ info] [filter:kubernetes:kubernetes.2] local POD info OK [2021/10/02 03:36:10] [ info] [filter:kubernetes:kubernetes.2] testing connectivity with API server... [2021/10/02 03:36:10] [ info] [filter:kubernetes:kubernetes.2] connectivity OK [2021/10/02 03:47:28] [ info] [filter:kubernetes:kubernetes.1] https=1 host=kubernetes.default.svc port=443 [2021/10/02 03:47:28] [ info] [filter:kubernetes:kubernetes.1] local POD info OK [2021/10/02 03:47:28] [ info] [filter:kubernetes:kubernetes.1] testing connectivity with API server... [2021/10/02 03:47:28] [ info] [filter:kubernetes:kubernetes.1] connectivity OK [2021/10/02 03:58:25] [ info] [filter:kubernetes:kubernetes.1] https=1 host=kubernetes.default.svc port=443 [2021/10/02 03:58:25] [ info] [filter:kubernetes:kubernetes.1] local POD info OK [2021/10/02 03:58:25] [ info] [filter:kubernetes:kubernetes.1] testing connectivity with API server... [2021/10/02 03:58:25] [ info] [filter:kubernetes:kubernetes.1] connectivity OK

There are quite a few error logs that mention the flogger pod logs: [2021/10/02 03:36:34] [error] [plugins/in_tail/tail_file.c:1289 errno=2] No such file or directory [2021/10/02 03:36:34] [error] [plugins/in_tail/tail_fs_inotify.c:139 errno=2] No such file or directory [2021/10/02 03:36:34] [error] [input:tail:tail.2] inode=352325017 cannot register file /var/log/containers/flogger-79fb55496c-wsck9_cat_flog-404e6754d7eeb58210e94e41eabc554faf8a70799182a7b57b61155704b9de56.log [2021/10/02 03:36:35] [error] [plugins/in_tail/tail_file.c:1289 errno=2] No such file or directory [2021/10/02 03:36:35] [error] [plugins/in_tail/tail_fs_inotify.c:139 errno=2] No such file or directory [2021/10/02 03:36:35] [error] [input:tail:tail.2] inode=352325073 cannot register file /var/log/containers/flogger-79fb55496c-wsck9_cat_flog-404e6754d7eeb58210e94e41eabc554faf8a70799182a7b57b61155704b9de56.log

Can anyone advise?

Oct 02 '21 05:10 awcheney

It looks like this in terms of error messages.https://github.com/fluent/fluent-bit/issues/4103 Can you show me the configuration of the Output plug-in?

Oct 08 '21 01:10 wenchajun