cp-ansible icon indicating copy to clipboard operation
cp-ansible copied to clipboard

Fixed Kafka Connect children registration with MDS

Open DennisFederico opened this issue 2 years ago • 0 comments

Description

Kafka Connect Cluster registration against MDS is not working properly when multiple kafka connect clusters are defined in the inventory (children). The registration is done only once and for the first cluster sub-group, moreover it includes all the hosts of the other groups too. I've followed the same approach of PR #991, with the addition of resolving the hostname using 'confluent.platform.resolve_hostname' in case 'hostname_aliasing_enabled: true'

Type of change

  • [X] Bug fix (non-breaking change which fixes an issue)
  • [ ] New feature (non-breaking change which adds functionality)
  • [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • [ ] This change requires a documentation update

How Has This Been Tested?

Using a connect an inventory with a kafka_connect similar to this one: ` kafka_connect: vars: hostname_aliasing_enabled: true children: connect-main: vars: kafka_connect_cluster_name: connect-main kafka_connect_group_id: connect-main kafka_connect_service_name: connect-main kafka_connect_config_filename: connect-main-distributed.properties kafka_connect_log_dir: "{{ kafka_connect_default_log_dir }}/connect-main" hosts: dfederico-demo-connect-0: dfederico-demo-connect-1:

connect-spawn1:
  vars:
    kafka_connect_cluster_name: connect-spawn1
    kafka_connect_group_id: connect-spawn1
    kafka_connect_service_name: connect-spawn1
    kafka_connect_config_filename: connect-spawn1-distributed.properties
    kafka_connect_log_dir: "{{ kafka_connect_default_log_dir }}/connect-spawn1"
  hosts:
    dfederico-demo-connect-2.A:
      ansible_host: dfederico-demo-connect-2
      hostname: dfederico-demo-connect-2
    dfederico-demo-connect-3.A:
      ansible_host: dfederico-demo-connect-3
      hostname: dfederico-demo-connect-3

connect-spawn2:
  vars:
    kafka_connect_cluster_name: connect-spawn2 
    kafka_connect_group_id: connect-spawn2
    kafka_connect_service_name: connect-spawn2
    kafka_connect_config_filename: connect-spawn2-distributed.properties
    kafka_connect_rest_port: 8084
    kafka_connect_jmxexporter_port: 8078
    kafka_connect_log_dir: "{{ kafka_connect_default_log_dir }}/connect-spawn2"
  hosts:
    dfederico-demo-connect-2.B:
      ansible_host: dfederico-demo-connect-2
      hostname: dfederico-demo-connect-2
    dfederico-demo-connect-3.B:
      ansible_host: dfederico-demo-connect-3
      hostname: dfederico-demo-connect-3

`

Without the change the registration end like this... Only one cluster gets registered with every "inventory" hostname (note the suffix) for every child cluster Screenshot 2022-06-08 at 18 26 33

With the fix it seems to work as expected... Each child cluster is registered with their intended host resolved to the real hostname and port (note spawn2) Screenshot 2022-06-08 at 20 27 03

Checklist:

  • [X] My code follows the style guidelines of this project
  • [X] I have performed a self-review of my own code
  • [X] I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [X] My changes generate no new warnings
  • [ ] I have added tests that prove my fix is effective or that my feature works
  • [ ] Any dependent changes have been merged and published in downstream modules
  • [ ] Any variable changes have been validated to be backwards compatible

DennisFederico avatar Jun 08 '22 18:06 DennisFederico