kafka icon indicating copy to clipboard operation
kafka copied to clipboard
verified publisher icon apache

KAFKA-17573: Move KRaftMetadataCache to metadata module

Open FrankYang0529 opened this issue 10 months ago • 5 comments

Changes

  • Move KRaftMetadataCache from core to metadata module
  • Change KRaftMetadataCache from Scala to Java

Performance

  • ReplicaFetcherThreadBenchmark 
    ./jmh-benchmarks/jmh.sh -f 1 -i 2 -wi 2

org.apache.kafka.jmh.fetcher.ReplicaFetcherThreadBenchmark

* trunk

Benchmark (partitionCount) Mode Cnt Score Error Units ReplicaFetcherThreadBenchmark.testFetcher 100 avgt 2 4669.047 ns/op ReplicaFetcherThreadBenchmark.testFetcher 500 avgt 2 24722.430 ns/op ReplicaFetcherThreadBenchmark.testFetcher 1000 avgt 2 57837.371 ns/op ReplicaFetcherThreadBenchmark.testFetcher 5000 avgt 2 485991.588 ns/op

* branch

Benchmark (partitionCount) Mode Cnt Score Error Units ReplicaFetcherThreadBenchmark.testFetcher 100 avgt 2 4581.417 ns/op ReplicaFetcherThreadBenchmark.testFetcher 500 avgt 2 24503.970 ns/op ReplicaFetcherThreadBenchmark.testFetcher 1000 avgt 2 56348.142 ns/op ReplicaFetcherThreadBenchmark.testFetcher 5000 avgt 2 470634.225 ns/op


* KRaftMetadataRequestBenchmark

./jmh-benchmarks/jmh.sh -f 1 -i 2 -wi 2 org.apache.kafka.jmh.metadata.KRaftMetadataRequestBenchmark

* trunk

Benchmark (partitionCount) (topicCount) Mode Cnt Score Error Units KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 10 500 avgt 2 943495.449 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 10 1000 avgt 2 1923336.954 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 10 5000 avgt 2 22469603.257 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 20 500 avgt 2 1564799.213 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 20 1000 avgt 2 3218482.586 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 20 5000 avgt 2 29796675.484 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 50 500 avgt 2 3604029.278 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 50 1000 avgt 2 7617695.388 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 50 5000 avgt 2 55070054.797 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 10 500 avgt 2 331.724 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 10 1000 avgt 2 337.840 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 10 5000 avgt 2 337.959 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 20 500 avgt 2 334.827 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 20 1000 avgt 2 342.852 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 20 5000 avgt 2 322.059 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 50 500 avgt 2 328.329 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 50 1000 avgt 2 336.541 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 50 5000 avgt 2 334.077 ns/op

* branch

Benchmark (partitionCount) (topicCount) Mode Cnt Score Error Units KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 10 500 avgt 2 671028.571 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 10 1000 avgt 2 1435193.244 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 10 5000 avgt 2 19727430.833 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 20 500 avgt 2 1114655.107 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 20 1000 avgt 2 2249700.266 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 20 5000 avgt 2 25811047.360 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 50 500 avgt 2 2311887.438 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 50 1000 avgt 2 4537162.770 ns/op KRaftMetadataRequestBenchmark.testMetadataRequestForAllTopics 50 5000 avgt 2 44013921.418 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 10 500 avgt 2 330.509 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 10 1000 avgt 2 337.044 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 10 5000 avgt 2 332.035 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 20 500 avgt 2 341.664 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 20 1000 avgt 2 322.385 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 20 5000 avgt 2 333.220 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 50 500 avgt 2 335.865 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 50 1000 avgt 2 326.180 ns/op KRaftMetadataRequestBenchmark.testRequestToJson 50 5000 avgt 2 336.809 ns/op


* PartitionMakeFollowerBenchmark

./jmh-benchmarks/jmh.sh -f 1 -i 2 -wi 2 org.apache.kafka.jmh.partition.PartitionMakeFollowerBenchmark

* trunk

Benchmark Mode Cnt Score Error Units PartitionMakeFollowerBenchmark.testMakeFollower avgt 2 158.161 ns/op

* branch

Benchmark Mode Cnt Score Error Units PartitionMakeFollowerBenchmark.testMakeFollower avgt 2 156.056 ns/op


* UpdateFollowerFetchStateBenchmark

./jmh-benchmarks/jmh.sh -f 1 -i 2 -wi 2 org.apache.kafka.jmh.partition.UpdateFollowerFetchStateBenchmark

* trunk

Benchmark Mode Cnt Score Error Units UpdateFollowerFetchStateBenchmark.updateFollowerFetchStateBench avgt 2 5006.529 ns/op UpdateFollowerFetchStateBenchmark.updateFollowerFetchStateBenchNoChange avgt 2 4900.634 ns/op

* branch

Benchmark Mode Cnt Score Error Units UpdateFollowerFetchStateBenchmark.updateFollowerFetchStateBench avgt 2 5031.341 ns/op UpdateFollowerFetchStateBenchmark.updateFollowerFetchStateBenchNoChange avgt 2 4987.916 ns/op


* CheckpointBench

./jmh-benchmarks/jmh.sh -f 1 -i 2 -wi 2 org.apache.kafka.jmh.server.CheckpointBench

* trunk

Benchmark (numPartitions) (numTopics) Mode Cnt Score Error Units CheckpointBench.measureCheckpointHighWatermarks 3 100 thrpt 2 0.927 ops/ms CheckpointBench.measureCheckpointHighWatermarks 3 1000 thrpt 2 0.678 ops/ms CheckpointBench.measureCheckpointHighWatermarks 3 2000 thrpt 2 0.489 ops/ms CheckpointBench.measureCheckpointLogStartOffsets 3 100 thrpt 2 0.998 ops/ms CheckpointBench.measureCheckpointLogStartOffsets 3 1000 thrpt 2 0.719 ops/ms CheckpointBench.measureCheckpointLogStartOffsets 3 2000 thrpt 2 0.572 ops/ms

* branch

Benchmark (numPartitions) (numTopics) Mode Cnt Score Error Units CheckpointBench.measureCheckpointHighWatermarks 3 100 thrpt 2 0.975 ops/ms CheckpointBench.measureCheckpointHighWatermarks 3 1000 thrpt 2 0.673 ops/ms CheckpointBench.measureCheckpointHighWatermarks 3 2000 thrpt 2 0.483 ops/ms CheckpointBench.measureCheckpointLogStartOffsets 3 100 thrpt 2 1.002 ops/ms CheckpointBench.measureCheckpointLogStartOffsets 3 1000 thrpt 2 0.718 ops/ms CheckpointBench.measureCheckpointLogStartOffsets 3 2000 thrpt 2 0.575 ops/ms


* PartitionCreationBench

./jmh-benchmarks/jmh.sh -f 1 -i 2 -wi 2 org.apache.kafka.jmh.server.PartitionCreationBench

* trunk

Benchmark (numPartitions) (useTopicIds) Mode Cnt Score Error Units PartitionCreationBench.makeFollower 20 false avgt 2 6.200 ms/op PartitionCreationBench.makeFollower 20 true avgt 2 7.244 ms/op

* branch

Benchmark (numPartitions) (useTopicIds) Mode Cnt Score Error Units PartitionCreationBench.makeFollower 20 false avgt 2 6.144 ms/op PartitionCreationBench.makeFollower 20 true avgt 2 7.169 ms/op

FrankYang0529 avatar Mar 18 '25 16:03 FrankYang0529

@chia7712, may you help to review this when you have time? This PR improves some performance like KRaftMetadataRequestBenchmark. I hope we can include this in 4.1. Thanks.

FrankYang0529 avatar Apr 14 '25 14:04 FrankYang0529

@FrankYang0529 please fix the conflicts

chia7712 avatar Apr 15 '25 12:04 chia7712

@chia7712, fixed conflicts Thanks.

FrankYang0529 avatar Apr 15 '25 13:04 FrankYang0529

@chia7712, I addressed all comments. Could you review again when you have time? Thanks.

FrankYang0529 avatar Apr 21 '25 15:04 FrankYang0529

Hi @chia7712, I addressed all comments. Could you review again when you have time? Thanks.

FrankYang0529 avatar May 20 '25 14:05 FrankYang0529

@FrankYang0529 could you please fix the conflicts?

chia7712 avatar Aug 17 '25 18:08 chia7712

@chia7712 I just fixed conflicts. Could you review again when you have time? Thanks.

FrankYang0529 avatar Aug 19 '25 06:08 FrankYang0529