Having issues with AWS MSK with TLS enabled

[rmarou]

Hi,

I am getting this when a producer contacts MSK through kafka-proxy with TLS.

INFO[2019-12-08T10:34:11Z] Reading data from local connection on 10.20.6.169:9092 from 10.20.12.42:40272 (b-1.msk-a.uv35cd.c3.kafka.eu-west-1.amazonaws.com:9094) had error: tls: received unexpected handshake message of type *tls.clientHelloMsg when waiting for *tls.clientKeyExchangeMsg

I ran the kafka-proxy with Client auth and without it.. I had the same problem, as long as I have TLS enabled with msk cluster.

./kafka-proxy server --proxy-listener-tls-enable --tls-enable --proxy-listener-key-file "ssl.key" --proxy-listener-cert-file "server-cert.pem" --debug-enable --log-level debug --bootstrap-server-mapping "b-1.msk-a.uv35cd.c3.kafka.eu-west-1.amazonaws.com:9094,0.0.0.0:9092,msk.a.sandbox.aws.corpinc.com:9092" --bootstrap-server-mapping "b-2.msk-a.uv35cd.c3.kafka.eu-west-1.amazonaws.com:9094,0.0.0.0:9093,msk.a.sandbox.aws.corpinc.com:9093" --bootstrap-server-mapping "b-3.msk-a.uv35cd.c3.kafka.eu-west-1.amazonaws.com:9094,0.0.0.0:9094,msk.a.sandbox.aws.corpinc.com:9094"

./kafka-console-producer.sh --broker-list grappler.msk.a.sandbox.aws.corpinc.com:9092 --topic tls.tested.test --producer.config client.properties

client.properties: security.protocol=SSL ssl.truststore.location=kafka.client.truststore.jks ssl.truststore.password=changeit

I am using current versions :

• For Kafka producer : kafka_2.12-2.2.1
• MSK with in-transit encryption enabled and 2.2.1 kafka version.

[pmbuko]

I'll soon be testing a similar setup. Don't you also need to specify --tls-ca-chain-cert-file /path/to/ca-chain.cert.pem or one of the other cert options?

EDIT:

I see you did specify ssl.key and server-cert.pem files, but that's for client auth, right?

[everesio]

I cannot reproduce your problem.

1. I created a mks cluster like in https://gist.github.com/everesio/262e11c6e5cebf56f1d5111c8cd7da3f but with some differences like

resource "aws_security_group" "kafka-proxy-cluster-security-group" {
  vpc_id = data.aws_vpc.vpc.id

  ingress {
    from_port   = 9094
    to_port     = 9094
    protocol    = "tcp"
    security_groups = [aws_security_group.kafka-proxy-security-group.id]
  }
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_msk_cluster" "kafka-proxy-cluster" {
  cluster_name           = "kafka-proxy-cluster"
  kafka_version          = "2.2.1"
  number_of_broker_nodes = 3

  broker_node_group_info {
    instance_type   = "kafka.m5.large"
    client_subnets  = [tolist(data.aws_subnet_ids.subnets.ids)[0],tolist(data.aws_subnet_ids.subnets.ids)[1],tolist(data.aws_subnet_ids.subnets.ids)[2]]
    security_groups = [aws_security_group.kafka-proxy-cluster-security-group.id]
    ebs_volume_size = 20
  }

  encryption_info {
    encryption_in_transit {
      client_broker = "TLS"
    }
  }

2. Created a certs and truststore with open SSL for kafka-proxy.xxx.info
3. Started a kafka-proxy (v0.1.6) on one ec2 instance with public IP having A record for kafka-proxy.xxx.info

kafka-proxy server \
--tls-enable \
--log-level debug \
--bootstrap-server-mapping "b-1.kafka-proxy-cluster.fei3vd.c2.kafka.eu-central-1.amazonaws.com:9094,0.0.0.0:32500,kafka-proxy.xxx.info:32500" \
--bootstrap-server-mapping "b-2.kafka-proxy-cluster.fei3vd.c2.kafka.eu-central-1.amazonaws.com:9094,0.0.0.0:32501,kafka-proxy.xxx.info:32501" \
--bootstrap-server-mapping "b-3.kafka-proxy-cluster.fei3vd.c2.kafka.eu-central-1.amazonaws.com:9094,0.0.0.0:32502,kafka-proxy.xxx.info:32502" \
--proxy-listener-tls-enable \
--proxy-listener-key-file ./ca-key.pem \
--proxy-listener-cert-file ./ca-cert.pem

4. From my linux desktop, I had no problem with following commands (kafka_2.12-2.3.0)

kafka-topics.sh --create --bootstrap-server kafka-proxy.xxx.info:32500,kafka-proxy.xxx.info:32501,kafka-proxy.xxx.info:3250 --replication-factor 3 --partitions 16 --topic test --command-config client-ssl.properties
kafka-console-producer.sh --broker-list kafka-proxy.xxx.info:32500,kafka-proxy.xxx.info:32501,kafka-proxy.xxx.info:32502 --topic test --producer.config client-ssl.properties

client-ssl.properties

security.protocol=SSL
ssl.truststore.location=/home/everesio/aws-mks-tls/certs/kafka-truststore.jks
ssl.truststore.password=mytruststorepass

[gustavomcarmo]

Hi @maru-podmast,

Have you solved the problem?

I'm getting the same tls: received unexpected handshake message of type *tls.clientHelloMsg when waiting for *tls.clientKeyExchangeMsg error.

If you've already solved, please share your solution here.

Thanks!

[everesio]

The error message looks like a https://github.com/golang/go/issues/36285

[rmarou]

I have given up at the time.. as it was critical to get solved for me. Will give it ago and let you know sometimes next week. Also, given you didn't reproduce it, feel free close it..

[gustavomcarmo]

Hi @everesio and @maru-podmast,

In my case the error went away after importing the CA root certificate and the CA intermediate certificate separatedly into the truststore, instead of importing from a single cert file with both certificates.

Thanks!

[amitkhurdhara1990]

@everesio Hi, we have a setup for Kafka 3 node cluster on azure with 3 centos vm and Kafka version is 3.0.0 hosted in EUW region. Ultimately we want to enable the Kafka Proxy to host in EUN region which will have the --bootstrap-server-mapping to EUW region hosted vm's.

But for testing when now in the current setup, where proxy and brokers are in same region and even in same vpc with port opened, I am getting below error .

./kafka-proxy server --debug-enable --bootstrap-server-mapping x.x.x.20:9094,x.x.x.11:19094 --log-level debug
--proxy-listener-key-file "dev-ssl-key" \ --proxy-listener-cert-file "dev-server.crt"
--proxy-listener-ca-chain-cert-file "dev-ca.cert"
--proxy-listener-tls-enable \ --sasl-enable --sasl-jaas-config-file ssl-client.properties output: INFO[2022-01-27T19:15:39Z] Starting kafka-proxy version 0.3.0 FATA[2022-01-27T19:15:39Z] tls: private key does not match public key

content of ssl-client.properties is...

security.protocol=SASL_SSL ssl.truststore.location=kafka-dev.server.truststore.jks ssl.truststore.password=changeit01 sasl.mechanism=PLAIN ssl.endpoint.identification.algorithm=BLANK sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="XXX" password="###"; Can you please help where I might be making mistake.