fluent-bit-kubernetes-logging icon indicating copy to clipboard operation
fluent-bit-kubernetes-logging copied to clipboard

Published 20 hours ago • verified publisher icon fluent

fluent-bit kafka data pipeline is not working.

Open bharathreddy901 opened this issue 2 years ago • 2 comments

Hi Team,

We have configured the fluent-bit->Kafka data pipeline with the below mentioned link.

https://github.com/fluent/fluent-bit-kubernetes-logging/tree/master/output/kafka

When we have deployed this pipeline, pods are coming up, Kafka topic was created, but logs are not flowing to Kafka cluster. We could not see any error in the pod logs, we are seeing below info. #########################################################

[root@centos]# kubectl logs fluent-bit-nb886 -n ec2-kafka Fluent Bit v1.5.7

  • Copyright (C) 2019-2020 The Fluent Bit Authors
  • Copyright (C) 2015-2018 Treasure Data
  • Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
  • https://fluentbit.io

[2022/09/12 07:11:40] [ info] [engine] started (pid=1) [2022/09/12 07:11:40] [ info] [storage] version=1.0.5, initializing... [2022/09/12 07:11:40] [ info] [storage] in-memory [2022/09/12 07:11:40] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128 [2022/09/12 07:11:40] [ info] [filter:kubernetes:kubernetes.0] https=1 host=kubernetes.default.svc port=443 [2022/09/12 07:11:40] [ info] [filter:kubernetes:kubernetes.0] local POD info OK [2022/09/12 07:11:40] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with API server... [2022/09/12 07:11:40] [ info] [filter:kubernetes:kubernetes.0] API server connectivity OK [2022/09/12 07:11:40] [ info] [output:kafka:kafka.0] brokers='x.x.x.x:9092,x.x.x.x:9092,x.x.x.x:9092' topics='kafka-bit' [2022/09/12 07:11:40] [ info] [sp] stream processor started ##########################################################################

Please let us know how to solve this issue.

Thanks in advance..!!

bharathreddy901 avatar Sep 12 '22 11:09 bharathreddy901

Hi Team - Please let me know if anyone resolved above problem.

bharathreddy901 avatar Sep 13 '22 03:09 bharathreddy901

Hi @edsiper and @hobti01,

Logs are not flowing from Fluent-bit to Kafka, But Kafka topic is getting created. We are using below link to configure the pipeline. We have not changed anything, only thig we have updated Kafka broker name & topic name.

https://github.com/fluent/fluent-bit-kubernetes-logging/tree/master/output/kafka

Thanks in advance.

bharathreddy901 avatar Sep 13 '22 11:09 bharathreddy901