jocko

jocko copied to clipboard

Alright so here's what I think would be cool to accomplish with Jocko:

v1: Kafka ~= Jocko

v1 is get Jocko on par with Kafka 0.10.x. People should be able to switch out Kafka for Jocko and run it in prod. I head up the architecture at an analytics company that runs terabytes of data through Kafka everyday and I'd switch it out for Jocko.

So:

• Implement Kafka in Go
  • With clean, easy to understand code.
  • Well tested.
  • Fast.
  • Just have to download a single binary to run it.

I've implemented Jocko so it understands Kafka's protocol. This way people can use Jocko with any client or service that works with Kafka. You'd have to make no code changes to switch from Kafka to Jocko.

Currently what's left is to implement consumer groups, replication, and benchmark/battle harden.

excellent work!

any plant for supporint push?

This is great. To clarify this..

I head up the architecture at an analytics company that runs terabytes of data through Kafka everyday and I'd switch it out for Jocko.

Did you do this or do you intend to? Also do you have a list of simple tasks that early adopters or supports can help with?

@WanliTian no. I generally don't want to be incompatible with Kafka's API for now and for this specifically, I agree with the reasoning to pull in Kafka's design documentation.

@bruth I intend to. As for contributing, consumer groups would be the simplest one I can think of and I've already added the protocols needed for that feature. But I need to add documentation and a contributing doc to help out with that but I'm focused on replication right now.

great

This is great & would be a boost for Go too! Salute!

Is it possible to use jocko for testing purpose right now? If yes could you provide a readme for how to start jocko?

@mlegenhausen I was trying to get it working tonight. Had to make some tweaks to work for me. Added a quick start guide in this PR https://github.com/travisjeffery/jocko/pull/10

Cleaned up the examples a bit: https://github.com/travisjeffery/jocko/tree/master/examples. Simple produce and fetch may work for you now given that sarama example works. You would have to create a topic like I have there — since clients don't support creating topics yet. A script for that would be a good contribution. @mlegenhausen I'd be interested to know what your outcome is, let me know in a new issue.

Any plans on changing High-Availability options?

I'm asking because I don't like Kafka's take on ISR and stuff, I'm more of a fan of Cassandra Consistency level approach with Quorum as a sweet spot between HA and Consistency guarantees.

@curlup probably you want to check uber/cherami-server.

Any plans for getting Samza to play nicely with this? Samza still wants zookeeper which means it causes issues trying to integrate the two.

Someone started on the ConsumerGroup impl? If yes, there are any design plans available?

I see not enough tests, one of the goals ("well tested") are in WIP I guess? Should help.

Any update?

Project's still going.

Wrote a doc on How you can contribute today.

Good work. I'm running the OpenFaaS project and working on a Kafka integration. Jocko could make development and testing much easier. I'm just trying to figure out whether we need consumer groups for MVP.

@travisjeffery Is there any plans to continue work on this now that you are working at Confluent? Or is progress being left solely up to the open source community? Just curious as I stumbled across this and it sounded like an interesting project to contribute to.

@brendanodwyer I plan to continue at some point. I'm currently writing a book (on building distributed services actually) and that takes up my free time. In any case I'll need some help and I'll help out any who are willing to put in the work.