pinot
pinot copied to clipboard
Upsert Compaction Generator: CRC mismatch robustness
During the UpsertCompactionTaskGenerator, we check whether the ZK metadata CRC of a segment matches the CRC we fetch from the validDocID bitmap response from the servers (Ref).
If a scenario is present where the CRCs do not match across replicas and we use the CRC from the validDocID bitmap response from a replica server that has not updated the ZK metadata, we will end up not considering that segment for compaction.
Tracking this issue for making this generation process more robust.