kop
kop copied to clipboard
[FEATURE] Use native instruction optimized CRC32C to improve batch.ensureValid()
Describe the solution you'd like
Use Crc32cLongChecksum
from bookkeeper
, instead of Crc32C.compute
. the Crc32cLongChecksum
uses SSE 4.2 to optimize the CRC32C algorithm, it might have 6 times better performance.
I do regular OMB tests of Pulsar clusters including with a version of KoP. We tend to get high end to end latency which appears to be more a function of the message size than it is the throughput. I looked at the flamegraphs from a flight recording and i noticed that a lot of time (over 20%) is spent in java.util.zip.CRC32C.update()
. I'm using OpenJDK 17.

@Demogorgon314 I'd like to upvote this "feature"!