Exposing Kafka outside EKS Kubernetes

[dsindatry]

Hi, I have successfully deployed kafka and zookeeper on EKS using cp-helm charts. I have exposed the kafka service using LoadBalancers. so that anyone outside the cluster can produce and consume messages. Any Idea how can I do it? This is what i tried

################################################################################################ $ kubectl get svc -n kafka NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kafka-0-external LoadBalancer 10.100.169.109 a6755784ac88e42abb0da08eb894282a-ae8ea84a5c931e66.elb.us-east-1.amazonaws.com 31090:31839/TCP 12h kafka-1-external LoadBalancer 10.100.92.249 ae2ae9b19a2744226933d2b851bf654d-db090f8c82cfe6df.elb.us-east-1.amazonaws.com 31090:31052/TCP 12h kafka-2-external LoadBalancer 10.100.184.132 a6eb334d9926742de81dc87eb950d48f-87513c070a4419ac.elb.us-east-1.amazonaws.com 31090:32395/TCP

################################################################################################ CONFIGURATION:

external: enabled: true type: LoadBalancer annotations: service.beta.kubernetes.io/aws-load-balancer-internal: "true" service.beta.kubernetes.io/aws-load-balancer-type: nlb

dns: useInternal: true useExternal: false distinct: false servicePort: 19092 firstListenerPort: 31090 domain: cluster.local loadBalancerIP: [] loadBalancerSourceRanges:

• "10.0.0.0/8"
• "10.240.0.0/12"
• "172.16.0.0/12"
• "192.0.0.0/8" init: image: "lwolf/kubectl_deployer" imageTag: "0.4" imagePullPolicy: "IfNotPresent"

"advertised.listeners": |- EXTERNAL://b${KAFKA_BROKER_ID}.cluster.local:31090

Uncomment to define the EXTERNAL Listener protocol

"listener.security.protocol.map": |- PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT

Anything else i need to do ? Has anyone tried this and can share some light ?

[aliadnani]

How are you trying to get data into Kafka? cp-helm-charts comes with a REST proxy instance, so you can expose that and use it to produce and consume messages. https://docs.confluent.io/current/kafka-rest/quickstart.html

[omkark95]

@dsindatry I have just completed it this week. :P I have modified the values.yaml for cp-kafka chart using the following:

region: <region> eg. ap-south-1

configurationOverrides:

# uncomment the following to expose it externally.
  "advertised.listeners": |-
   EXTERNAL://${HOST_IP}:$((31090 + ${KAFKA_BROKER_ID}))
  "listener.security.protocol.map": |-
   PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT

externaldns:
  dns: <DNS-HOSTED-ZONE>
  ttl: 60

nodeport:
  enabled: true
  servicePort: 19092
  firstListenerPort: 31090

loadbalancer:
  enabled: true
  type: nlb
  internal: true 
  loadbalancerPort: 9092
  backendprotocol: tcp
  tlspolicy: ELBSecurityPolicy-TLS-1-2-2017-01

ssl:
  certificateARN: <AWS-ARN>

I have also create a new file nlb-loadbalancer-service.yml in the templates folder of cp-kafka. nlb.zip

I have used external-dns to add the of loadbalancer dns to route53.

helm install external-dns \
   --set provider=aws \
   --set aws.zoneType=private \
   --set txtOwnerId=<hosted-zone-identifier> \
   --set domainFilters[0]=<hosted-zone-dns> \
   bitnami/external-dns

[MEOWMEOW114]

@dsindatry I have just completed it this week. :P I have modified the values.yaml for cp-kafka chart using the following:

region: <region> eg. ap-south-1

configurationOverrides:

# uncomment the following to expose it externally.
  "advertised.listeners": |-
   EXTERNAL://${HOST_IP}:$((31090 + ${KAFKA_BROKER_ID}))
  "listener.security.protocol.map": |-
   PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT

externaldns:
  dns: <DNS-HOSTED-ZONE>
  ttl: 60

nodeport:
  enabled: true
  servicePort: 19092
  firstListenerPort: 31090

loadbalancer:
  enabled: true
  type: nlb
  internal: true 
  loadbalancerPort: 9092
  backendprotocol: tcp
  tlspolicy: ELBSecurityPolicy-TLS-1-2-2017-01

ssl:
  certificateARN: <AWS-ARN>

I have also create a new file nlb-loadbalancer-service.yml in the templates folder of cp-kafka. nlb.zip

I have used external-dns to add the of loadbalancer dns to route53.

helm install external-dns \
   --set provider=aws \
   --set aws.zoneType=private \
   --set txtOwnerId=<hosted-zone-identifier> \
   --set domainFilters[0]=<hosted-zone-dns> \
   bitnami/external-dns

I tried your config yet it seems does not work I test using kafkacat -b {{ template "cp-kafka.name" $root }}-1.{{ $region }}.{{ $dns }} - L

is there other config in configurationOverrides necessary

[emmaroberts-nbs]

I couldn't get the nlb to work either.

[vjvel]

LB creation is fine, how to do the external advertise mapping to the LB name for the kafka brokers? @dsindatry