Support kafka 4.0 [breaking change]

[rukai]

closes https://github.com/tychedelia/kafka-protocol-rs/issues/114 closes https://github.com/tychedelia/kafka-protocol-rs/issues/54 - All raft/zk specific messages are removed in 4.0

kafka 4.0 removes a few versions/message types.

This results in a few complications.

1.

The fields field is now optional, so I marked it as #[serde(default)] to set it to an empty list instead of panicking when the field isnt present.

2.

To retain support for older protocols, we could in theory do some hacky thing with combining the json from multiple kafka versions. But I think it is reasonable for kafka-protocol to just drop support for the types/versions dropped by kafka 4.

I was previously concerned that the library would have unbounded growth in compile time and runtime as new versions and types keep being added. However, if the kafka team will remove old types/versions eventually, then we dont have to think about doing that ourselves which is great.

Additionally, the removed types are:

• UpdateMetadata - interbroker communication only
• StopReplica - interbroker communication only
• LeaderAndIsr - interbroker communication only
• ControlledShutdown - not very useful

Sources:

• This paragraph https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225153708#KIP866ZooKeepertoKRaftMigration-ControllerRPCs
• This field containing only "zkBroker" https://github.com/apache/kafka/blob/e3f953483cb480631bf041698770b47ddb82796f/clients/src/main/resources/common/message/LeaderAndIsrRequest.json#L19

So we haven't actually lost any meaningful messages at all.

We have however lost support for v0 and v1 records. https://cwiki.apache.org/confluence/display/KAFKA/KIP-724%3A+Drop+support+for+message+formats+v0+and+v1 Since this is tied to the produce requests version 0-3, which are removed, we cant even support v0 and v1 records in older clients/servers. So we should entirely remove v0 and v1 support in this PR.

I actually dont know how much v0 and v1 are used in the wild. But it might be worth doing a kafka-protocol v0.15 before releasing this PR as kafka-protocol v0.16

@michaelbeaumont would you want a release before this is merged?

[michaelbeaumont]

One thing I see in the KIP is:

We will remove write support for message formats v0 and v1 in Apache Kafka 4.0. Consumers will continue to support message formats v0 and v1. Similarly, brokers may still return message formats v0 and v1 with fetch v4 or higher if the relevant messages were originally written as v0/v1 and were not converted to v2.

So will we still potentially see v0/v1 in newer api versions?

[rukai]

Similarly, brokers may still return message formats v0 and v1 with fetch v4 or higher if the relevant messages were originally written as v0/v1 and were not converted to v2.

That is a great catch, thankyou. I only checked the producer.

We should continue supporting v0/v1 for consuming.

[rukai]

I looked into this further.

Turns out that even kafka 3.0 never stores records as v0/v1. If it receives v0/v1 it will convert up to v2 If it receives a fetch request on a protocol level that only supports v0/v1 it will down convert from v2 stored on disk.

This makes it impossible to test v0/v1 in kafka 3.0 on a 4.0 based kafka-protocol. The oldest fetch request we have available now (v4) will only return v0/v1 messages if they already exist on disk. And v0/v1 messages would only exist in kafka 3.0 if they were written to disk by a previous, very very old version of kafka.

I dont feel comfortable with code that we cannot test so I think we are better off just deleting the v0/v1 support.

However while kafka 3.0 is easy to run and confirm that these conversions take place. I've not had much luck with getting kafka 2.0 to run, so I have to assume its not converting to v2 versions before writing to disk.

3.0 has only been out since 2021, which is not that long. As such I think we should definitely do a final release (0.15.0) before merging this PR.

[tychedelia]

I think it's fine to keep our support pretty current. If someone comes along and wants to add support for older versions that's something we'd obvious accept but let's not make things overly complicated just on the basis of supporting 2.0.

[rukai]

I'm in no rush to see this in a release, but this PR is ready for review and merge.

[rukai]

I've done a bit of cleanup, @tychedelia can you please review this when you get the chance. (theres actually not much changes when you ignore the generated stuff)

[tychedelia]

Also @rukai I've just added you as an owner on crates.io. I've definitely moved significantly away from distributed computing space into doing Bevy work almost full time. I'd love to stay up to date on things but given the critical dependency for your company on this crate I wanted to give you full ability to do releases as needed. So I'd loved to be tagged on stuff but don't block on me in the future. :)

[rukai]

Thsnks!