automq icon indicating copy to clipboard operation
automq copied to clipboard

[Enhancement] Enhance KVRecord

Open superhx opened this issue 9 months ago • 4 comments

Who is this for and what problem do they have today?

AutoMQ team

Why is solving this problem impactful?

Currently, the following types of data are stored in the KVRecord:

  • Partition -> MetaStreamId: kafka$clusterId/$topicName/$partitionIndex -> $metaStreamId
  • Node -> NodeMetadata: __automq_node/$nodeId -> $nodeMetadata.

KVs of different functions are isolated by the prefix of the key.

Expectations:

  • Add namespace to KVRecord to more elegantly isolate (future) KVs used by different functions.
  • Add epoch to KVRecord to avoid concurrent operations of PutKVRequest accidentally overwriting KVs.

Additional notes

superhx avatar Mar 12 '25 03:03 superhx

I couldn't find KVRecord in the AutoMQ source code, is it part of AutoMQ or its dependencies?

yyyCode avatar Mar 15 '25 16:03 yyyCode

Hi team 👋,

I’d like to take this on if it's still unassigned.

I’ll investigate the reassignment path and ensure it gracefully handles the case where a partition has no segments (e.g., by skipping or initializing default state).

I’ll also add a test to cover this edge case. Let me know if there are any preferred conventions or context I should keep in mind!

Best,
Amin

AminMortezaie avatar May 20 '25 11:05 AminMortezaie

I've added my merge request.

AminMortezaie avatar May 20 '25 16:05 AminMortezaie

@AminMortezaie Thanks for your contribution! I'll take a look at it within a few days.

1sonofqiu avatar May 21 '25 04:05 1sonofqiu