in_kafka plugin

[cainelli]

Hi all, I was wondering if there is any plans to support input Kafka as done with out_kafka?

[ymartincds]

Is it for technical reason that there is out and not in Kafka binding ?

Regards

[kinghuang]

Now that fluent-but 1.1 has stream processing, I’m super interested in the ability to use Kafka topic(s) as input. It’d be a good alternative to KSQL for certain tasks.

At the moment, I’m using kafkacat (also built on librdkafka) to consume messages and pipe them to fluent-bit.

[marcosflobo]

Long time since this thread was opened. Any news on this? Is there any plan to support Kafka as input for FluentBit?

[tarunwadhwa13]

We too forward raw logs and use logstash for parsing and storing to ES. Since Logstash consumes too much resources, planning to move to fluent bit for this. But unfortunately, couldn't find any In Kafka support. Is there any workaround to achieve this?

[edsiper]

Yes, it will be implemented during Q3-Q4 this year.

[tarunwadhwa13]

Gentle Reminder !! Can we add this to 1.6 milestone?

[edsiper]

good idea. Added.

[tarunwadhwa13]

@edsiper We are eagerly waiting for this :smile: Is there expected date for 1.6 release?

[edsiper]

Moved to v1.7 release (Feb 2021)

[danielpcox]

Did this make it into the 1.7 release? I can't find it in the 1.7 documentation.

[edsiper]

It didn't make it yet :/

[agup006]

in_kafka is not in 1.7.0, the aim is for it to be in the 1.7.x series. This one has been long time coming :)

@danielpcox would you be open to testing when we have a prototype?

[danielpcox]

I'm game

[brandonganem]

Piling on here, this would be super helpful!

[brandonganem]

@agup006 I'm also happy to test a prototype.

[stephanmiehe]

Would be great to have this

[smandava]

Please add kafka input.

[agup006]

Hey folks, We definitely understand the need for in_kafka - We are trying to juggle a few priorities to get this going. for background and transparency, we are quite busy these next few month with FluentCon as well as better multiline support.

Looking for creative ways on how to expedite this, I know it's not the best update but appreciate everyone engaging and volunteering for testing

[hafkaM]

Hi. Some news? I can easily test some beta versions.

[hafkaM]

Hi. Some news?:)

[agup006]

Heya, some updates from our community meeting July 29th (https://docs.google.com/document/d/1vJvsn8E0SanLO1R0X3RC1qTw0XQK_7q75sZ8IbWAu-g/edit#heading=h.nowdn7qwlldd) was a member taking a look on adding it along with better rdkafka support. I'm not sure where this stands though happy to discuss at our next community meeting next Thursday if folks are interested

[hafkaM]

Hello. Any news on kafka input processor? I stop hoping it will ever be available. Thanks.

[agup006]

Hey folks, we needed to add some new core features in Fluent Bit input side to get an initial version - kudos to @tarruda for taking it on. Here is a poc if folks have some cycles to give it a spin

https://github.com/fluent/fluent-bit/pull/4641

[mcauto]

When will it be released?

[hari819]

Do we have plans to release this support , i am looking for an alternative to logstash and I find fluentbit is the one, Please do we have some plans ?

Thanks, Hari

[sheperdsonbrown]

@patrick-stephens I believe this PoC was merged to the main branch in February https://github.com/fluent/fluent-bit/pull/4641#issuecomment-1021739380 (here it is in code: https://github.com/fluent/fluent-bit/tree/master/plugins/in_kafka) I do not see any documentation for the input here: https://docs.fluentbit.io/manual/pipeline/inputs

[patrick-stephens]

I'm afraid I'm not in the loop on the Kafka side, @lecaros may know more on the docs side.

[conwayok]

Looking forward for this to be supported! Would be a great alternative to Logstash.

[coredump17]

Agree. This would be a great addition

[hafkaM]

Hi. Is there any estimate when it could be at least in some beta version including documentation?