Production deployments for no-kafka

[goodfella1408]

I am exploring kafka clients for node . The one we are using currently just isnt working for us (after a lot of wrappers and hooks to restart the client on errors).Are there any large scale prod deployments that are using no-kafka without any babysitting/issues . no-kafka is relatively new and not listed in - https://cwiki.apache.org/confluence/display/KAFKA/Clients so just wanted to ask .

[mohitkhanna]

+1

Thanks for asking this question. Even we are looking for some information on this.

Due to lack of production info on Node packages for Kafka we were forced to turn to Confluent's Kafka distribution (v0.10). They have a rest proxy for accessing Kafka over HTTP - which kinda worked for us. It's quite stable but has its learning curve and intricacies.

We would love to go to the native driver of we see more testimonials about production usage - as that would simplify our tech stack.

Regards, Mohit

On 01-Oct-2016, at 13:40, goodfella1408 [email protected] wrote:

I am exploring kafka clients for node . The one we are using currently just isnt working for us (after a lot of wrappers and hooks to restart the client on errors).Are there any large scale prod deployments that are using no-kafka without any babysitting/issues . no-kafka is relatively new and not listed in - https://cwiki.apache.org/confluence/display/KAFKA/Clients so just wanted to ask .

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[raviten]

Hi All, We are using no-kafka lib for only consuming data and it is quiet satisfactory for us. Our objective was to read the data from broker and pass to a function.

We are using 3 machine cluster of kafka Reading 2 topics with 4 partitions each and deployed 4 consumers(2 for each topic) in total.

Thanks, Ravi

[davidsketchdeck]

We've been using no-kafka in production over the last few weeks and having good success! I'd recommend writing a solid little test suite to ensure the library works in your production environment, that helped us iron out some kinks and have the confidence to release.

[ghost]

@raviten , @davidsketchdeck, @oleksiyk Greetings! Would you mind sharing your experience working with no-kafka module in production? Could you please tell me: do you use plaintext protocol for your kafka-consumers, and what is the approximate amount of messages being consumed from topic per second? And whether you use standard consumer settings, or some custom ones?

I am currently having trouble trying to use kafka client written in node.js to consume 3-5k messages from ssl-protected kafka cluster with 3 nodes and topic which have 3 partitions.

I had no success using node-rdkafka and kafka-node, and ended up using java client, piping out messages to stdout. So, if you have experience working with no-kafka in similar environments or even larger, would you mind sharing?

Looking forward to your reply.