TANK

TANK copied to clipboard

Merge-join multiple streams

1

It should be easy to implement a TankClient utility method that facilitates joining multiple streams based on the message timestamp and a simple merge strategy. For 1+ streams, we can...

markpapadakis

Consumer Offsets, support via internal topic/partition

1

According to @miguno ( I need to go through the Kafka codebase to figure out what they do exactly ), they are maintaining an internal/special-purpose topic `__consumer_offsets`, where the policy...

markpapadakis

Create a simple asciinema.org presentation for TANK, for the README file.

markpapadakis

There is partial support for wide index records already, and in practice, it won't be needed unless you have really HUGE datasets, but we should implement it anyway. This should...

markpapadakis

The Tank client, for performance, will use an internal to a connection buffer for the consumed messages. That is, instead of allocating memory to hold each consumed message, it simply...

markpapadakis

Consider batching incoming partition messages before flushing them together to partition segment

2

Currently, the broker will append to the leader (itself in standalone mode), but maybe this is not optimal. We should support for tuning select topics so that the server would...

markpapadakis

Mirror-Maker functionality for tank-cli

3

It should be pretty straight-forward to add support for mirroring from one Tank (broker|cluster) to another, by implementing mirroring functionality on tank-cli, similar to [Kafka's Mirror-Maker](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330). That is say, we...

markpapadakis

We currently have no real need for those features, but eventually we 'd like to offer robust encryption and authentication. See [Apache Kafka Security 101](http://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption)

markpapadakis

Compaction/cleanup improvements

1

Cleanup works fine, but we need to improve it with throttling support, and by exposing some hard coded options as configuration options. - [ ] Throttling. We are now reading...

markpapadakis

REST API

1

Check [Kafka REST](http://docs.confluent.io/2.0.0/kafka-rest/docs/intro.html) for ideas. Will be accessible over HTTP/1 and HTTP/2. Also check [Google's Pub/Sub REST API](https://cloud.google.com/pubsub/reference/rest/)

markpapadakis