connect icon indicating copy to clipboard operation
connect copied to clipboard

Refactor kafka components to reduce duplication

Open Jeffail opened this issue 1 year ago • 0 comments

I've done a broad refactor of the various kafka input/output components in order to:

  1. Reduce the amount of duplicate field definitions and parsers
  2. Reduce the amount of duplicate client code around applying those field definitions
  3. Improve our mechanisms for sharing client and connection info across components

There's quite a lot going on here and it touches lots of the existing connectors. The integration tests run fine but we'll need to do some manual testing of the migrator components before merging/releasing.

One facet of this work is the idea of bringing out an implementation of a "consumer" and "producer" into a reusable type, this is important as we will have multiple plugins essentially doing the same thing (reading or writing benthos messages from/to kafka records) where the implementation is non-trivial (ordering, checkpointing, batching).

Eventually I want to have a solid mechanism for instantiating inputs and outputs that will be more user friendly for guaranteeing things like strict ordering. However, this is a complicated body of work and so it's important to first ensure that it will only need to be implemeted in one place, and that the codebase is tidy enough to reason about.

Jeffail avatar Oct 07 '24 09:10 Jeffail