Add basic SASL and TLS support for Kafka cloud events
Tracking issue
Closes https://github.com/flyteorg/flyte/issues/3220
Why are the changes needed?
These changes are needed to use cloud events with kafka brokers with password or certificate auth enabled.
What changes were proposed in this pull request?
This change expands the KafkaConfig struct to include some SASL fields for password based auth as well as a TLSConfig struct for certificate based auth. It then provides a helper function to load these into the Sarama config. Lastly it adds helm support for loading the cloud events config from a secret since password based auth will contain sensitive fields.
How was this patch tested?
This was tested on our production Flyte clusters.
Check all the applicable boxes
- [x] I updated the documentation accordingly.
- [x] All new and existing tests passed.
- [x] All commits are signed-off.
Codecov Report
Attention: Patch coverage is 11.11111% with 32 lines in your changes missing coverage. Please review.
Project coverage is 36.83%. Comparing base (
553a702) to head (6d7a27f). Report is 143 commits behind head on master.
| Files with missing lines | Patch % | Lines |
|---|---|---|
| ...kg/runtime/interfaces/application_configuration.go | 8.57% | 29 Missing and 3 partials :warning: |
Additional details and impacted files
@@ Coverage Diff @@
## master #5814 +/- ##
==========================================
- Coverage 36.84% 36.83% -0.02%
==========================================
Files 1309 1309
Lines 130967 130998 +31
==========================================
- Hits 48259 48255 -4
- Misses 78524 78557 +33
- Partials 4184 4186 +2
| Flag | Coverage Δ | |
|---|---|---|
| unittests-datacatalog | 51.58% <ø> (ø) |
|
| unittests-flyteadmin | 54.06% <11.11%> (-0.09%) |
:arrow_down: |
| unittests-flytecopilot | 11.73% <ø> (ø) |
|
| unittests-flytectl | 62.40% <ø> (ø) |
|
| unittests-flyteidl | 6.92% <ø> (ø) |
|
| unittests-flyteplugins | 53.64% <ø> (ø) |
|
| unittests-flytepropeller | 43.00% <ø> (ø) |
|
| unittests-flytestdlib | 55.41% <ø> (ø) |
Flags with carried forward coverage won't be shown. Click here to find out more.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.