KAFKA-17023: add PCollectionsImmutableMap to ConcurrentMapBenchmark

[TaiJuWu]

*More detailed description of your change, Add SingleWriteMultiReadBenchmark for ConcurrentHashMap, CopyOnWriteMap, PcollectionsImmutableMap

Running Command: ./jmh-benchmarks/jmh.sh org.apache.kafka.jmh.util.SingleWriteMultiReadBenchmar

Test result:

Benchmark                                                                                     (mapSize)  (writePercentage)  Mode  Cnt      Score       Error  Units
SingleWriteMultiReadBenchmark.ConcurrentHashMap                                                     100                0.1  avgt    5   5551.856 ±  5184.034  ns/op
SingleWriteMultiReadBenchmark.ConcurrentHashMap:testConcurrentHashMapEntry                          100                0.1  avgt    5  12631.454 ± 11665.343  ns/op
SingleWriteMultiReadBenchmark.ConcurrentHashMap:testConcurrentHashMapGet                            100                0.1  avgt    5     50.020 ±    51.794  ns/op
SingleWriteMultiReadBenchmark.ConcurrentHashMap:testConcurrentHashMapRandomGet                      100                0.1  avgt    5     82.140 ±   111.144  ns/op
SingleWriteMultiReadBenchmark.ConcurrentHashMap:testConcurrentHashMapValues                         100                0.1  avgt    5   9998.994 ±  9485.352  ns/op
SingleWriteMultiReadBenchmark.ConcurrentHashMap:testConcurrentHashMapWrite                          100                0.1  avgt    5     ≈ 10⁻⁴              ns/op
SingleWriteMultiReadBenchmark.CopyOnWriteMap                                                        100                0.1  avgt    5   3571.190 ±  2335.478  ns/op
SingleWriteMultiReadBenchmark.CopyOnWriteMap:testCopyOnWriteMapEntry                                100                0.1  avgt    5   8741.245 ±  5825.625  ns/op
SingleWriteMultiReadBenchmark.CopyOnWriteMap:testCopyOnWriteMapGet                                  100                0.1  avgt    5     43.997 ±    27.091  ns/op
SingleWriteMultiReadBenchmark.CopyOnWriteMap:testCopyOnWriteMapRandomGet                            100                0.1  avgt    5     72.952 ±    69.792  ns/op
SingleWriteMultiReadBenchmark.CopyOnWriteMap:testCopyOnWriteMapValues                               100                0.1  avgt    5   5783.686 ±  3656.257  ns/op
SingleWriteMultiReadBenchmark.CopyOnWriteMap:testCopyOnWriteMapWrite                                100                0.1  avgt    5      0.001 ±     0.008  ns/op
SingleWriteMultiReadBenchmark.PcollectionsImmutableMap                                              100                0.1  avgt    5   5529.729 ±  1322.872  ns/op
SingleWriteMultiReadBenchmark.PcollectionsImmutableMap:testPcollectionsImmutableMapEntry            100                0.1  avgt    5   9151.719 ±  3652.206  ns/op
SingleWriteMultiReadBenchmark.PcollectionsImmutableMap:testPcollectionsImmutableMapGet              100                0.1  avgt    5    202.152 ±    85.313  ns/op
SingleWriteMultiReadBenchmark.PcollectionsImmutableMap:testPcollectionsImmutableMapRandomGet        100                0.1  avgt    5    320.565 ±    89.311  ns/op
SingleWriteMultiReadBenchmark.PcollectionsImmutableMap:testPcollectionsImmutableMapValues           100                0.1  avgt    5  12997.451 ±  2356.417  ns/op
SingleWriteMultiReadBenchmark.PcollectionsImmutableMap:testPcollectionsImmutableMapWrite            100                0.1  avgt    5      0.001 ±     0.004  ns/op

*Summary of testing strategy (including rationale)

Committer Checklist (excluded from commit message)

• [ ] Verify design and implementation
• [ ] Verify test coverage and CI build status
• [ ] Verify documentation (including upgrade notes)

[chia7712]

@TaiJuWu It seems copyOnWrite has better performance overall?

[TaiJuWu]

@TaiJuWu It seems copyOnWrite has better performance overall?

Yes, using copyOnWrite can get better read performance than others according the benchmark.

[chia7712]

Yes, using copyOnWrite can get better performance than others according the benchmark.

please take a look at the use case of acl cache. that will be a good start

[chia7712]

@TaiJuWu could you please add summary to description ?

[TaiJuWu]

@TaiJuWu could you please add summary to description ?

Hi @chia7712 , update to the top post, PTAL. Thanks a lot.

[chia7712]

@TaiJuWu could you please merge #16463 into this one? it would be great that we can see the benefit on the real use cases

[TaiJuWu]

After run existing benchmark, the result show

Benchmark                                        (aclCount)  (authorizerType)  (denyPercentage)  (resourceCount)  Mode  Cnt   Score    Error  Units (Patch)|(Unpatch)  Score    Error  Units
AuthorizerBenchmark.testAclsIterator                     10               ACL                20             5000  avgt   15   1.528 ±  0.014  ms/op (Patch)|(Unpatch)  1.677 ±  0.030  ms/op
AuthorizerBenchmark.testAclsIterator                     10               ACL                20            10000  avgt   15   5.343 ±  0.207  ms/op (Patch)|(Unpatch)  6.355 ±  0.158  ms/op
AuthorizerBenchmark.testAclsIterator                     10             KRAFT                20             5000  avgt   15   2.457 ±  0.015  ms/op (Patch)|(Unpatch)  2.858 ±  0.060  ms/op
AuthorizerBenchmark.testAclsIterator                     10             KRAFT                20            10000  avgt   15   5.722 ±  0.084  ms/op (Patch)|(Unpatch)  7.529 ±  0.173  ms/op
AuthorizerBenchmark.testAclsIterator                     20               ACL                20             5000  avgt   15   5.514 ±  0.180  ms/op (Patch)|(Unpatch)  6.175 ±  0.169  ms/op
AuthorizerBenchmark.testAclsIterator                     20               ACL                20            10000  avgt   15  20.672 ±  0.334  ms/op (Patch)|(Unpatch) 17.564 ±  0.071  ms/op
AuthorizerBenchmark.testAclsIterator                     20             KRAFT                20             5000  avgt   15   9.638 ±  7.254  ms/op (Patch)|(Unpatch)  6.725 ±  0.188  ms/op
AuthorizerBenchmark.testAclsIterator                     20             KRAFT                20            10000  avgt   15  12.769 ±  0.207  ms/op (Patch)|(Unpatch) 16.018 ±  0.118  ms/op

AuthorizerBenchmark.testAuthorizeByResourceType          10               ACL                20             5000  avgt   15   0.007 ±  0.001  ms/op (Patch)|(Unpatch)  0.007 ±  0.001  ms/op
AuthorizerBenchmark.testAuthorizeByResourceType          10               ACL                20            10000  avgt   15   0.007 ±  0.001  ms/op (Patch)|(Unpatch) 17.412 ±  0.074  ms/op
AuthorizerBenchmark.testAuthorizeByResourceType          10             KRAFT                20             5000  avgt   15  17.104 ±  0.267  ms/op (Patch)|(Unpatch) 14.330 ±  0.229  ms/op
AuthorizerBenchmark.testAuthorizeByResourceType          10             KRAFT                20            10000  avgt   15  25.149 ±  0.412  ms/op (Patch)|(Unpatch) 29.884 ±  2.134  ms/op
AuthorizerBenchmark.testAuthorizeByResourceType          20               ACL                20             5000  avgt   15   0.007 ±  0.001  ms/op (Patch)|(Unpatch)  0.007 ±  0.001  ms/op
AuthorizerBenchmark.testAuthorizeByResourceType          20               ACL                20            10000  avgt   15   0.007 ±  0.001  ms/op (Patch)|(Unpatch)  0.007 ±  0.001  ms/op
AuthorizerBenchmark.testAuthorizeByResourceType          20             KRAFT                20             5000  avgt   15  24.736 ±  0.407  ms/op (Patch)|(Unpatch) 30.647 ±  0.213  ms/op
AuthorizerBenchmark.testAuthorizeByResourceType          20             KRAFT                20            10000  avgt   15  46.468 ±  0.409  ms/op (Patch)|(Unpatch) 53.107 ±  0.282  ms/op

AuthorizerBenchmark.testAuthorizer                       10               ACL                20             5000  avgt   15   0.107 ±  0.001  ms/op (Patch)|(Unpatch)  0.076 ±  0.001  ms/op
AuthorizerBenchmark.testAuthorizer                       10               ACL                20            10000  avgt   15   0.130 ±  0.001  ms/op (Patch)|(Unpatch)  0.122 ±  0.001  ms/op
AuthorizerBenchmark.testAuthorizer                       10             KRAFT                20             5000  avgt   15   0.270 ±  0.002  ms/op (Patch)|(Unpatch)  0.269 ±  0.001  ms/op
AuthorizerBenchmark.testAuthorizer                       10             KRAFT                20            10000  avgt   15   1.562 ±  1.101  ms/op (Patch)|(Unpatch)  0.362 ±  0.002  ms/op
AuthorizerBenchmark.testAuthorizer                       20               ACL                20             5000  avgt   15   0.481 ±  0.070  ms/op (Patch)|(Unpatch)  0.080 ±  0.001  ms/op
AuthorizerBenchmark.testAuthorizer                       20               ACL                20            10000  avgt   15   0.724 ±  0.112  ms/op (Patch)|(Unpatch)  0.116 ±  0.001  ms/op
AuthorizerBenchmark.testAuthorizer                       20             KRAFT                20             5000  avgt   15   0.309 ±  0.004  ms/op (Patch)|(Unpatch)  0.267 ±  0.001  ms/op
AuthorizerBenchmark.testAuthorizer                       20             KRAFT                20            10000  avgt   15   0.415 ±  0.004  ms/op (Patch)|(Unpatch)  0.377 ±  0.001  ms/op

AuthorizerBenchmark.testUpdateCache                      10               ACL                20             5000  avgt   15  15.881 ±  0.162  ms/op (Patch)|(Unpatch) 16.032 ±  0.097  ms/op
AuthorizerBenchmark.testUpdateCache                      10               ACL                20            10000  avgt   15  35.924 ±  0.468  ms/op (Patch)|(Unpatch) 31.811 ±  0.159  ms/op
AuthorizerBenchmark.testUpdateCache                      10             KRAFT                20             5000  avgt   15  ≈ 10⁻⁶           ms/op (Patch)|(Unpatch) ≈ 10⁻⁶           ms/op
AuthorizerBenchmark.testUpdateCache                      10             KRAFT                20            10000  avgt   15  ≈ 10⁻⁶           ms/op (Patch)|(Unpatch) ≈ 10⁻⁶           ms/op
AuthorizerBenchmark.testUpdateCache                      20               ACL                20             5000  avgt   15  31.230 ±  1.868  ms/op (Patch)|(Unpatch) 26.076 ±  0.139  ms/op
AuthorizerBenchmark.testUpdateCache                      20               ACL                20            10000  avgt   15  60.179 ±  0.381  ms/op (Patch)|(Unpatch) 55.851 ±  0.290  ms/op
AuthorizerBenchmark.testUpdateCache                      20             KRAFT                20             5000  avgt   15  ≈ 10⁻⁶           ms/op (Patch)|(Unpatch) ≈ 10⁻⁶           ms/op
AuthorizerBenchmark.testUpdateCache                      20             KRAFT                20            10000  avgt   15  ≈ 10⁻⁶           ms/op (Patch)|(Unpatch) ≈ 10⁻⁶           ms/op
JMH benchmarks done

In general, there is no obvious benefit from these benchmark, maybe we need to dig it deep.

[chia7712]

In general, there is no obvious benefit from these benchmark, maybe we need to dig it deep.

thanks for this sharing. Please feel free to ping me if there is any interesting finding.

[github-actions[bot]]

This PR is being marked as stale since it has not had any activity in 90 days. If you would like to keep this PR alive, please leave a comment asking for a review. If the PR has merge conflicts, update it with the latest from the base branch.

If you are having difficulty finding a reviewer, please reach out on the [mailing list](https://kafka.apache.org/contact).

If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed.

[github-actions[bot]]

This PR has been closed since it has not had any activity in 120 days. If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open the PR and ask for a review.