kop icon indicating copy to clipboard operation
kop copied to clipboard

[FEATURE] Use native instruction optimized CRC32C to improve batch.ensureValid()

Open Demogorgon314 opened this issue 3 years ago • 2 comments

Describe the solution you'd like Use Crc32cLongChecksum from bookkeeper, instead of Crc32C.compute. the Crc32cLongChecksum uses SSE 4.2 to optimize the CRC32C algorithm, it might have 6 times better performance.

Demogorgon314 avatar Dec 16 '21 02:12 Demogorgon314

I do regular OMB tests of Pulsar clusters including with a version of KoP. We tend to get high end to end latency which appears to be more a function of the message size than it is the throughput. I looked at the flamegraphs from a flight recording and i noticed that a lot of time (over 20%) is spent in java.util.zip.CRC32C.update(). I'm using OpenJDK 17.

Screen Shot 2022-10-31 at 12 36 56 PM

dave2wave avatar Oct 31 '22 21:10 dave2wave

@Demogorgon314 I'd like to upvote this "feature"!

dave2wave avatar Oct 31 '22 21:10 dave2wave