kafka-go

kafka-go copied to clipboard

Based on #855, this package adds two new primitives to support indexing Kafka partitions by message keys: records.Index and records.Partition.

records.Index is an interface intended to serve as an abstraction layer for indexing message keys to their offset in Kafka.

records.Partition combines the index, cache, and storage to provide the high-level API for inserting and looking up key/value pairs in Kafka partitions.

I have completed most of the work on records.Index, adding a SQLite-backed implementation as example tho this would not necessarily be the backend of choice in production.

This is still a work in progress tho, the records.Partition part is still incomplete and being fleshed out.

Among other changes I've made:

• the kafka-go/records package is a separate module to allow depending on Go 1.18 instead of being bound to backward compatiblity with 1.15 that kafka-go requires
• I'm taking a dependency on github.com/segmentio/datastructures as it was hard to find a solid balanced tree implementation to implement the indexes