kafka icon indicating copy to clipboard operation
kafka copied to clipboard

Port tools for topic configuration

Open lucasbru opened this issue 4 months ago • 0 comments

Ports several tools for topic configuration from the client side, to the broker-side. Several things are refactored:

  • Decoupling. On the client side, for example, RepartitionTopics was using the CopartitionedTopicEnforcer, the InternalTopicCreator, the TopologyMetadata and the Cluster objects. All of those are mocked with Mockito during testing. This points to bad coupling. We refactored all classes to be mostly self-sufficient, only relying on themselves and simple interfaces.
  • Tests only use JUnit5, not hamcrast matchers and no other streams utilities, to not pollute the group coordinator module.
  • All classes only modify the configurations -- the code does not actually call into the AdminClient anymore.
  • We map all errors to new errors in the broker, in particular, the error for missing topics, inconsistent internal topics, and invalid topologies.

We include the internal, mutable datastructures, that are set- and map-based for effiecient algorithms. They are distinctly different from the data represented in StreamsGroupTopologyValue and StreamsGroupInitializeRequest, since regular expressions must be resolved at this point.

Both the topic creation and internal topic validation will be based on this code.

Every time, either the broker-side topology or the topic metadata on the broker changes, we reconfigure the internal topics, check consistency with the current topics on the broker, and possibly trigger creation of the missing internal topics. These changes will be built on top of this change.

Committer Checklist (excluded from commit message)

  • [ ] Verify design and implementation
  • [ ] Verify test coverage and CI build status
  • [ ] Verify documentation (including upgrade notes)

lucasbru avatar Oct 04 '24 13:10 lucasbru