kafka
kafka copied to clipboard
apache
KAFKA-15599: Move KafkaMetadataLog to the raft module
- Rename
KafkaMetadataLogtoKafkaRaftLog - Make
raftdepend onstorage - Remove
storagedependency onmetadataas this otherwise create a cyclic dependency
There are some concerns regarding this change, though we haven't made kraft a public API, we are trying to keep it dependency-free to be used as a raft-sdk anywhere. For example, if you are implementing a system with metadata changing frequently, you can directly copy kraft code into your project.
This assumption also has some benefits to the whole Kafka architecture, this change reduced dependency complexity but breaked this assumption, we should evaluate which goal is more important.
we are trying to keep it dependency-free to be used as a raft-sdk anywhere
Where has this been discussed?
The way Kafka implements Raft is via a topic so our implementation of ReplicatedLog needs to depend on storage. Since metadata depends on raft, the other option is to put KafkaMetadataLog into the metadata module and have that depend on storage instead of raft.
Where has this been discussed?
In KIP-595 we wrote:
We are also trying to pave the way for normal partition replication through Raft as well as eventually supporting metadata sharding with multiple partitions.
We are only trying to pave the way for Kafka normal partition replication instead of making it SDK exportable, I may have thought too much about it, but the idea behind it still deserves consideration, for example, etcd-raft(https://github.com/etcd-io/raft) also follows a similar principle to not include a concrete log implementation.
Moving it to metadata module is better compared putting it in raft module, and we should not rename it to KafkaRaftLog since we can use raft to replicate normal partition.
I'm not sure I interpret that statement from KIP-595 as an intent to keep the raft module dependency free. As you said the raft module is not public API, and I've not seen any discussions in that direction.
The metadata module mostly consists of the data structures written to the metadata log. To me it seems the logic of maintaining the metadata log as a topic fits better in the raft module, as described in https://issues.apache.org/jira/browse/KAFKA-15599.
@mimaison That makes sense to me, 👍 if we should make the tradeoff, the modularity outweighs the pluggability of the raft module, @jsancio @ijuma, where do you think is better to put KafkaMetadataLog.
The description I wrote in the ticket was the conclusion after a discussion with a few people including @jsancio and @cmccabe. @jsancio does that approach still make sense to you?
@jsancio @junrao I keep rebasing this each week. Quick reminder, can you take another look? Thanks