tls option for AWS MSK

[davi020]

I am using AWS MSK with SSL enabled and I need to know how to create the tls cert and key file from keystore. We are using keystore to connect to AWS MSK.

tls burrow config:

[tls.name]
certfile="/etc/kafkakeys/orderhistory/client_cert.pem"
keyfile="/etc/kafkakeys/orderhistory/private_key.pem"
cafile="/etc/kafkakeys/orderhistory/truststore.pem"
noverify=false

Logs:

Reading configuration from /etc/burrow
{"level":"info","ts":1646032362.8147514,"msg":"Started Burrow"}
{"level":"info","ts":1646032362.814817,"msg":"configuring","type":"coordinator","name":"zookeeper"}
{"level":"info","ts":1646032362.8156712,"msg":"configuring","type":"coordinator","name":"storage"}
{"level":"info","ts":1646032362.8157008,"msg":"configuring","type":"module","coordinator":"storage","class":"inmemory","name":"mystorage"}
{"level":"info","ts":1646032362.8162286,"msg":"configuring","type":"coordinator","name":"evaluator"}
{"level":"info","ts":1646032362.8162525,"msg":"configuring","type":"module","coordinator":"evaluator","class":"caching","name":"default"}
{"level":"info","ts":1646032362.8162856,"msg":"configuring","type":"coordinator","name":"httpserver"}
{"level":"info","ts":1646032362.8164155,"msg":"configuring","type":"coordinator","name":"notifier"}
{"level":"info","ts":1646032363.0167787,"msg":"configuring","type":"coordinator","name":"cluster"}
{"level":"info","ts":1646032363.016819,"msg":"configuring","type":"module","coordinator":"cluster","class":"kafka","name":"orderhistory"}
{"level":"panic","ts":1646032363.0298462,"msg":"cannot read TLS certificate or key file: tls: failed to parse private key"}
panic: cannot read TLS certificate or key file: tls: failed to parse private key [recovered]
	panic: cannot read TLS certificate or key file: tls: failed to parse private key [recovered]
	panic: cannot read TLS certificate or key file: tls: failed to parse private key

goroutine 1 [running]:
main.handleExit()
	/usr/src/Burrow/main.go:64 +0x21d
panic(0xc673e0, 0xc0004614f0)
	/usr/local/go/src/runtime/panic.go:969 +0x175
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0000dd760, 0x0, 0x0, 0x0)
	/go/pkg/mod/go.uber.org/[email protected]/zapcore/entry.go:230 +0x55f
go.uber.org/zap.(*Logger).Panic(0xc00007a240, 0xc0003b6b40, 0x49, 0x0, 0x0, 0x0)
	/go/pkg/mod/go.uber.org/[email protected]/logger.go:225 +0x85
github.com/linkedin/Burrow/core.configureCoordinators.func1(0xc00007a060)
	/usr/src/Burrow/core/burrow.go:97 +0x7d
panic(0xc673e0, 0xc0004614e0)
	/usr/local/go/src/runtime/panic.go:969 +0x175
github.com/linkedin/Burrow/core/internal/helpers.GetSaramaConfigFromClientProfile(0xc0001e5bb0, 0xc, 0x23)
	/usr/src/Burrow/core/internal/helpers/sarama.go:115 +0xf57
github.com/linkedin/Burrow/core/internal/cluster.(*KafkaCluster).Configure(0xc00028eb00, 0xc00022fd60, 0xc, 0xc0001aa740, 0x14)
	/usr/src/Burrow/core/internal/cluster/kafka_cluster.go:63 +0x133
github.com/linkedin/Burrow/core/internal/cluster.(*Coordinator).Configure(0xc00000e340)
	/usr/src/Burrow/core/internal/cluster/coordinator.go:83 +0x22b
github.com/linkedin/Burrow/core.configureCoordinators(0xc00007a060, 0xf25d00, 0xc000078100, 0xf25cc0, 0xc000032420, 0xf25a80, 0xc00000e2c0, 0xf25b80, 0xc000032510, 0xf25bc0, ...)
	/usr/src/Burrow/core/burrow.go:104 +0xc3
github.com/linkedin/Burrow/core.Start(0xc00007a060, 0xc0000bf800, 0x0)
	/usr/src/Burrow/core/burrow.go:152 +0x418
main.main()
	/usr/src/Burrow/main.go:115 +0x695

Related Issue : [https://github.com/Shopify/sarama/issues/1799] can someone point me to a doc?

[riletan]

What is the type of your keystore? If you are using the Amazon RootCAs, then there is no private key in your keystore, so you need to use insecure config like bellow, no certfile or keyfile is required, it will not work if you declare these files in tls config. [tls.name] cafile="/path/to/msk.client.ca" noverify=true

You can crawl the cert form cluster domain name using openssl : openssl s_client -showcerts -servername <clusterurl> -connect <clusterurl>:<port> </dev/null > msk.client.ca then edit the generated file, delete anything outside of -----BEGIN CERTIFICATE-----/-----END CERTIFICATE-----

Or if you have truststore.jks file, you can generate the ca cert file using keytool:

keytool -exportcert -alias $ALIAS -keystore truststore.jks -storepass $PAWSSW -rfc -file msk.client.ca