flyte icon indicating copy to clipboard operation
flyte copied to clipboard

Add basic SASL and TLS support for Kafka cloud events

Open Sovietaced opened this issue 1 year ago • 1 comments

Tracking issue

Closes https://github.com/flyteorg/flyte/issues/3220

Why are the changes needed?

These changes are needed to use cloud events with kafka brokers with password or certificate auth enabled.

What changes were proposed in this pull request?

This change expands the KafkaConfig struct to include some SASL fields for password based auth as well as a TLSConfig struct for certificate based auth. It then provides a helper function to load these into the Sarama config. Lastly it adds helm support for loading the cloud events config from a secret since password based auth will contain sensitive fields.

How was this patch tested?

This was tested on our production Flyte clusters.

Check all the applicable boxes

  • [x] I updated the documentation accordingly.
  • [x] All new and existing tests passed.
  • [x] All commits are signed-off.

Sovietaced avatar Oct 06 '24 00:10 Sovietaced

Codecov Report

Attention: Patch coverage is 11.11111% with 32 lines in your changes missing coverage. Please review.

Project coverage is 36.83%. Comparing base (553a702) to head (6d7a27f). Report is 143 commits behind head on master.

Files with missing lines Patch % Lines
...kg/runtime/interfaces/application_configuration.go 8.57% 29 Missing and 3 partials :warning:
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5814      +/-   ##
==========================================
- Coverage   36.84%   36.83%   -0.02%     
==========================================
  Files        1309     1309              
  Lines      130967   130998      +31     
==========================================
- Hits        48259    48255       -4     
- Misses      78524    78557      +33     
- Partials     4184     4186       +2     
Flag Coverage Δ
unittests-datacatalog 51.58% <ø> (ø)
unittests-flyteadmin 54.06% <11.11%> (-0.09%) :arrow_down:
unittests-flytecopilot 11.73% <ø> (ø)
unittests-flytectl 62.40% <ø> (ø)
unittests-flyteidl 6.92% <ø> (ø)
unittests-flyteplugins 53.64% <ø> (ø)
unittests-flytepropeller 43.00% <ø> (ø)
unittests-flytestdlib 55.41% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Oct 06 '24 00:10 codecov[bot]