pinot icon indicating copy to clipboard operation
pinot copied to clipboard

Add Referencing feature in TLS config to reduce the duplicated rows

Open dongxiaoman opened this issue 3 years ago • 3 comments

Description

Adds reference feature to TLS config loading. Now we can use xxxx.tls.ref=<name> to simply ask TLSUtils to load TLS settings from the same config file.

  • It is introduced to reduce the duplicated entries in some cases like PinotBroker.
  • It is fully backwards compatible The old settings will still work; we just provide a new way to add config

Why do we need it

Right now if Broker is running inside a Kubernetes cluster, we may need 3 or 4 sets of TLS settings for it: broker https, broker gRPC secure, broker client.access.internal, and client.access.external . It makes the config file super long with duplicated entries. With this PR we can simply write TLS settings once, and reference it when we need it.

Tests done

Added several unit tests to cover it.

Release Notes

  • TLS settings in config can be set by using format like xxx.tls.ref=<name> to reference a shared setting set by pinot.shared.tls.<name>

dongxiaoman avatar Aug 22 '22 00:08 dongxiaoman

cc @apucher this will shorten broker config files a lot. Please have a look, thanks!

dongxiaoman avatar Aug 22 '22 01:08 dongxiaoman

Codecov Report

Merging #9264 (13b363c) into master (718f41f) will decrease coverage by 7.26%. The diff coverage is 5.88%.

:exclamation: Current head 13b363c differs from pull request most recent head c2cd41b. Consider uploading reports for the commit c2cd41b to get more accurate results

@@             Coverage Diff              @@
##             master    #9264      +/-   ##
============================================
- Coverage     68.75%   61.48%   -7.27%     
+ Complexity     4755     4538     -217     
============================================
  Files          1859     1847      -12     
  Lines         99129    98782     -347     
  Branches      15077    15040      -37     
============================================
- Hits          68161    60741    -7420     
- Misses        26076    33447    +7371     
+ Partials       4892     4594     -298     
Flag Coverage Δ
integration1 26.40% <5.88%> (-0.03%) :arrow_down:
unittests1 67.07% <ø> (-0.01%) :arrow_down:
unittests2 ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...apache/pinot/broker/api/HttpRequesterIdentity.java 28.57% <0.00%> (-57.15%) :arrow_down:
...org/apache/pinot/broker/api/RequesterIdentity.java 50.00% <0.00%> (-50.00%) :arrow_down:
...va/org/apache/pinot/spi/utils/CommonConstants.java 27.69% <ø> (ø)
...roker/requesthandler/BaseBrokerRequestHandler.java 56.59% <50.00%> (-10.87%) :arrow_down:
...ot/broker/requesthandler/BrokerRequestHandler.java 0.00% <0.00%> (-100.00%) :arrow_down:
...pinot/controller/recommender/io/ConfigManager.java 0.00% <0.00%> (-100.00%) :arrow_down:
.../org/apache/pinot/client/AggregationResultSet.java 0.00% <0.00%> (-100.00%) :arrow_down:
...ntroller/recommender/rules/impl/JsonIndexRule.java 0.00% <0.00%> (-100.00%) :arrow_down:
...troller/recommender/io/metadata/FieldMetadata.java 0.00% <0.00%> (-100.00%) :arrow_down:
...roller/recommender/rules/impl/BloomFilterRule.java 0.00% <0.00%> (-100.00%) :arrow_down:
... and 261 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

codecov-commenter avatar Aug 22 '22 01:08 codecov-commenter

we sort-of have this ability already. the default TLS settings for each component (broker, controller, ...) should already be injected in every connector-specific config.

See for example, here: https://github.com/apache/pinot/blob/13b363c109e69d5069cc41490cc9ebc37078093a/pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/TlsIntegrationTest.java#L134. All the default properties, like controller.tls.keystore.type are automatically copied into the internal and external connector configs

apucher avatar Aug 22 '22 17:08 apucher