Feature ideas

[travisjeffery]

trafficstars

This issue is where we can propose and collaborate on feature ideas.

When you submit an idea, describe the problem and propose a solution.

How you can contribute today.

[ghost]

I would like to play with using on it, mobiles etc. A good logging, tracing basis is needed when you distribute over these options style devices. There are so many ways to do logging and tracing. The main thing it to log out today cloud. I don't want to suggest a solution because it's too opinionated. I just want to point out it's importance to slow faster innovation cycles

[choleraehyq]

Is there any benchmark between kafka and jocko?

[elvarb]

Authentication and transport encryption is essential. Especially if the authentication management could be done better than it is in Kafka right now.

[kylemcc]

You may already be thinking along these lines, but here are a few things I'd like to see:

1. An HTTP-based admin API could be handy. This would enable a several useful things:
   • Easily building admin features directly into the CLI: E.g.: jocko admin create-topic -name=topic ...
   • A rich admin UI (a la kafka-manager)
   • All kinds of third-party tooling
2. In reference to point 1, it'd be great to actually build admin functions and a ui directly into jocko. I think consul and vault by Hashicorp are a great example of this - shipping one binary that includes the server, admin ui, and admin CLI.
3. Would love to see a metrics API exported. My personal vote would be to support Prometheus directly by integrating their Go library

In addition to everything you're already doing, I think think this would go a long way in making Jocko a more user/ops-friendly Kafka alternative.

I'm just starting to dig into the code, but I'm open to discussing specific ideas and helping where I can.

[thedrow]

What about the Kafka Connect API? It's very useful for streaming changes from one data source to another.

[ghost]

@kylemcc i agree on the 3 points as being the low hanging fruit.

• gui admin
• infrastructure like etc integration. consul & vault are good candidates
  • curious about security, which was not mentioned.
• metrics and logging
  • prom is a great one since its golang and heavily backed (like integration into grpc)
  • metrics ?

[elvarb]

@thedrow another way would be to go a similar way as nsq does it. They have a separate tool called nsq_to_nsq to handle this. Makes the server itself simpler.

[skotchpine]

@kylemcc, @gedw99 is it possible to build a wrapper for the 'core' jocko that includes fancy gui tooling?

[skotchpine]

@choleraehyq does Kafka have a benchmarking script or strategy we can build from?

[travisjeffery]

@skotchpine checkout https://github.com/apache/kafka/blob/trunk/bin/kafka-producer-perf-test.sh/https://github.com/apache/kafka/blob/trunk/tools/src/main/java/org/apache/kafka/tools/ProducerPerformance.java and its consumer counterpart.

[skotchpine]

Damn, they have an obscene amount of bash written