pinot icon indicating copy to clipboard operation
pinot copied to clipboard

Upsert Compaction Generator: CRC mismatch robustness

Open tibrewalpratik17 opened this issue 1 year ago • 0 comments

During the UpsertCompactionTaskGenerator, we check whether the ZK metadata CRC of a segment matches the CRC we fetch from the validDocID bitmap response from the servers (Ref).

If a scenario is present where the CRCs do not match across replicas and we use the CRC from the validDocID bitmap response from a replica server that has not updated the ZK metadata, we will end up not considering that segment for compaction.

Tracking this issue for making this generation process more robust.

tibrewalpratik17 avatar Jun 27 '24 05:06 tibrewalpratik17