kafka icon indicating copy to clipboard operation
kafka copied to clipboard

KafkaProxy Type

Open steve-gray opened this issue 7 years ago • 0 comments

Here's some thinking:

  • Add a new class 'KafkaProxy' (note, to avoid conflict with future ES6 proxy, prefixed name).
  • Proxy would act as a single, stateful bridge to a whole Kafka cluster - allowing clients to be able to speak to horizontally scaled Kafka instances whilst only needing to actually communicate with a single endpoint.
  • Clients would connect with a raw, unmodified Kafka client (i.e. no-kafka, or any other client) - and we'd basically remap and redirect the real requests under the hood in accordance with the current leaders/brokers.

One of two ways I can see this happening:

  1. Just add it to the library, sits alongside the 4 primary types (Simple, GroupConsumer, Producer and GroupAdmin).
  2. Externalize all the protocol stuff into a kafka-protocol package, then I can just write the proxy against that.

I think #2 is semantically the best, but realistically it'd just make evolving no-kafka much harder. Secret option #3 is a just borrow the code directly into another package or start cross-reaching for internal files with a require(), but that means suddenly the internals of no-kafka become a contract - and that's pretty ugly too.

I'm happy to basically bang this out with some tests tonight/during the week.

steve-gray avatar Sep 12 '16 09:09 steve-gray