DataflowTemplates icon indicating copy to clipboard operation
DataflowTemplates copied to clipboard

Add support for SSL in Kafka to BigQuery template.

Open pranavbhandari24 opened this issue 1 year ago • 2 comments

After this PR is merged, customers should be able to pass in the SSL configs for the kafka consumer by following the below steps,

  1. Save the truststore.jks and keystore.jks files in a GCS bucket (e.g, test-bucket)
  2. Create a file and paste the following,
{
   "bucket":"test-bucket",
   "ssl.truststore.location":"truststore.jks",
   "ssl.keystore.location":"keystore.jks",
   "ssl.truststore.password":"truststore-password",
   "ssl.keystore.password":"keystore-password",
   "ssl.key.password":"admin-secret"
}
  1. Replace the values in the JSON. Note, ssl.truststore.location and ssl.keystore.location should be the path of the truststore and keystore respectively under the bucket (In this example, the truststore is stored in gs://test-bucket/truststore.jks)
  2. Omit any key-value pairs that are not applicable.
  3. Create a Secret manager secret using that file. (https://cloud.google.com/secret-manager)
  4. Pass the Secret Manager secret ID using the parameter kafkaConfigSecretId

pranavbhandari24 avatar Sep 18 '23 20:09 pranavbhandari24

How do you pass "security.protocol=SASL_SSL" and SCRAM configs for Kafka here?

nishkarshv avatar Sep 22 '23 06:09 nishkarshv

Any update on this PR? I have a project where I need to import data from an SSL secured kafka server and was hoping to use the dataflow template.

mathnut02 avatar Feb 15 '24 16:02 mathnut02

This pull request has been marked as stale due to 180 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time. Thank you for your contributions.

github-actions[bot] avatar Aug 14 '24 02:08 github-actions[bot]