Prozess
Prozess copied to clipboard
cainus
direct zookeeper integration
Maybe 'zookeeper integration' is not an objective itself but rather the functionality that it would provide, such as automatic broker discovery and broker reconnection.
The Kafka driver design docs outline the specific responsibilities of Zookeeper within the Kafka driver. I think that just saying "Zookeeper integration" is a broad term used to outline them all but for the sake of github issues we could break them down into their specific tasks.
It's not quite ready for a pull request, but I'm working actively on a Zookeeper client implementation for enumerating brokers and storing consumer group offsets. It's a few small TODO's from being basically functional (see the README for my initial use-case). Your feedback/suggestions are welcome.
https://github.com/devoncrouse/Prozess
Awesome. :)
If I was going to do this, I'd start it as a new node package/project on its own (one that uses Prozess). You can still slap "Prozess" into the name if you want (or not), and we'd probably be able to contribute too. There are a few reasons that that makes a better factoring, most importantly being the fact that there are still some use-cases for lower-level clients, so I'd hate to abstract away the ability to store offsets elsewhere.
In general though I've been sold on the node aesthetic of keeping tiny libraries that build on top of other libraries. For instance, I'm almost certainly about to build a KafkaReadableStream, and that'll be a new package entirely (one that uses Prozess). KnowWhatIMean?
I can help you out with starting a new project/package too, if you've never done that before (package files, travis integration, etc). I'd love to see you slay this dragon, because it's something we'll need sooner or later ourselves.
What zookeeper client are you integrating? I was originally looking at this client https://github.com/yfinkelstein/node-zookeeper
I can definitely do that. Might ping you for questions on Travis when that time comes, but TDD didn't happen for this one and that time will be...later :)
Yes, yfinkelstein/node-zookeeper is the one. Not using znode watches yet, but I plan to.
Did you npm install it? I don't see it listed in package.json. Thanks for getting into this by the way
On Wed, Jul 17, 2013 at 10:32 AM, Devon Crouse [email protected]:
Yes, yfinkelstein/node-zookeeper is the one. Not using znode watches yet, but I plan to.
— Reply to this email directly or view it on GitHubhttps://github.com/cainus/Prozess/issues/21#issuecomment-21129776 .
You're right; I haven't added the dependency yet. And very happy to help - the Kafka clients of the world need some love...
https://github.com/devoncrouse/node-kafka-zookeeper
I'll keep working on it; last night I got it consuming from all brokers/partitions, but there are still bugs. I'm guessing we can close this issue if Zk is no longer on the road map for Prozess. Contribution/feedback on node-kafka-zookeeper is more than welcome, of course.
I guess that seems like the right thing to do. We will want to note it in the Prozess README and point people to the Zookeeper Consumer.
On Wed, Jul 17, 2013 at 11:06 AM, Devon Crouse [email protected]:
https://github.com/devoncrouse/node-kafka-zookeeper
I'll keep working on it; last night I got it consuming from all brokers/partitions, but there are still bugs. I'm guessing we can close this issue if Zk is no longer on the road map for Prozess. Contribution/feedback on node-kafka-zookeeper is more than welcome, of course.
— Reply to this email directly or view it on GitHubhttps://github.com/cainus/Prozess/issues/21#issuecomment-21132235 .
What about using https://github.com/adobe-research/libkafka for supporting 0.8 or the C library https://github.com/edenhill/librdkafka might be better? The offsets to be managed could be handed over to the implementer through a callback function and let the node app manage it. this would be a cool feature. could also support something built in so folks dont have to implement something if they need something that just works out of the box to start. i have been considering whipping up a rest service for this type of thing, could have the client to that service post the changes to zookeeper... so integrate zookeeper through a rest interface with a C end point. any interest?
