confluent.ssl playbook updates only broker's jks ignoring all others.

[ay-b]

We're playing with Kafka on-premise and using LetsEncrypt to generate certificates. All variables for custom certificates were set up and initial deployment worked as intended. Recently LE-cert has expired and we need to update jks-bundles. So I've copied LE cert-files and used confluent.ssl playbook to generate new bundles. However, this playbook did update only for the broker jks-bundle, ignoring other components and even zookeeper. Running minimal and connect deployments playbooks fixed the situation, but is extremely inconvenient due to the prolonged service outage.

Steps to reproduce the behavior:

• Generate LE wildcard certs
• Set all variables for custom certs in role
• Deploy Kafka Minimal+Connect
• Wait for certs to expire
• Run confluent_ssl playbook

Expected behaviour Kafka brokers, schema and connect jks-bundles being regenerated, services restarted and everything is up again

Inventory File

---
all:
  vars:
    ansible_connection: ssh
    ansible_user: superuser
    ansible_port: 32329
    ansible_become: true
    ansible_python_interpreter: /usr/bin/python3
    sasl_protocol: plain
    zookeeper_sasl_protocol: digest
    ssl_enabled: true
    ssl_custom_certs: true
    ssl_key_filepath: "/etc/letsencrypt/live/domain.name/privkey.pem"
    regenerate_keystore_and_truststore: true
    jmxexporter_enabled: true
    confluent_server_enabled: false
    admin_secret: secret
    client_secret: secret
    schema_registry_secret: secret
    kafka_connect_secret: secret
    kafka_rest_secret: secret
    ksql_secret: secret
    control_center_secret: secret
    kafka_connect_replicator_secret: secret
    kafka_broker_secret: secret
    mds_super_user_password: secret
    schema_registry_ldap_password: secret
    kafka_connect_ldap_password: secret
    kafka_rest_ldap_password: secret
    control_center_ldap_password: secret
    kafka_connect_replicator_ldap_password: secret
    _sasl_scram_users: "{
      'admin': {
        'principal': 'admin',
        'password': '{{admin_secret}}'
      },
      'client': {
        'principal': 'client',
        'password': '{{client_secret}}'
      }{% if 'schema_registry' in groups %},
      'schema_registry': {
        'principal': 'schema_registry',
        'password': '{{schema_registry_secret}}'
      }{% endif %}{% if 'kafka_connect' in groups %},
      'kafka_connect': {
        'principal': 'kafka_connect',
        'password': '{{kafka_connect_secret}}'
      }{% endif %}{% if 'kafka_rest' in groups %},
      'kafka_rest': {
        'principal': 'kafka_rest',
        'password': '{{kafka_rest_secret}}'
      }{% endif %}{% if 'ksql' in groups %},
      'ksql': {
        'principal': 'ksql',
        'password': '{{ksql_secret}}'
      }{% endif %}{% if 'control_center' in groups %},
      'control_center': {
        'principal': 'control_center',
        'password': '{{control_center_secret}}'
      }{% endif %}{% if 'kafka_connect_replicator' in groups %},
      'kafka_connect_replicator': {
        'principal': 'kafka_connect_replicator',
        'password': '{{kafka_connect_replicator_secret}}'
      }{% endif %}
    }"
    sasl_scram_users: {}
    _sasl_scram256_users: "{
      'admin': {
        'principal': 'admin',
        'password': '{{admin_secret}}'
      },
      'client': {
        'principal': 'client',
        'password': '{{client_secret}}'
      }{% if 'schema_registry' in groups %},
      'schema_registry': {
        'principal': 'schema_registry',
        'password': '{{schema_registry_secret}}'
      }{% endif %}{% if 'kafka_connect' in groups %},
      'kafka_connect': {
        'principal': 'kafka_connect',
        'password': '{{kafka_connect_secret}}'
      }{% endif %}{% if 'kafka_rest' in groups %},
      'kafka_rest': {
        'principal': 'kafka_rest',
        'password': '{{kafka_rest_secret}}'
      }{% endif %}{% if 'ksql' in groups %},
      'ksql': {
        'principal': 'ksql',
        'password': '{{ksql_secret}}'
      }{% endif %}{% if 'control_center' in groups %},
      'control_center': {
        'principal': 'control_center',
        'password': '{{control_center_secret}}'
      }{% endif %}{% if 'kafka_connect_replicator' in groups %},
      'kafka_connect_replicator': {
        'principal': 'kafka_connect_replicator',
        'password': '{{kafka_connect_replicator_secret}}'
      }{% endif %}
    }"
    sasl_scram256_users: {}
    _sasl_plain_users: "{
      'admin': {
        'principal': 'admin',
        'password': '{{admin_secret}}'
      },
      'client': {
        'principal': 'client',
        'password': '{{client_secret}}'
      }{% if 'schema_registry' in groups %},
      'schema_registry': {
        'principal': '{% if ccloud_kafka_enabled|bool %}{{ccloud_kafka_key}}{% else %}schema_registry{% endif %}',
        'password': '{% if ccloud_kafka_enabled|bool %}{{ccloud_kafka_secret}}{% else %}{{schema_registry_secret}}{% endif %}'
      }{% endif %}{% if 'kafka_connect' in groups %},
      'kafka_connect': {
        'principal': '{% if ccloud_kafka_enabled|bool %}{{ccloud_kafka_key}}{% else %}kafka_connect{% endif %}',
        'password': '{% if ccloud_kafka_enabled|bool %}{{ccloud_kafka_secret}}{% else %}{{kafka_connect_secret}}{% endif %}'
      }{% endif %}{% if 'kafka_rest' in groups %},
      'kafka_rest': {
        'principal': '{% if ccloud_kafka_enabled|bool %}{{ccloud_kafka_key}}{% else %}kafka_rest{% endif %}',
        'password': '{% if ccloud_kafka_enabled|bool %}{{ccloud_kafka_secret}}{% else %}{{kafka_rest_secret}}{% endif %}'
      }{% endif %}{% if 'ksql' in groups %},
      'ksql': {
        'principal': '{% if ccloud_kafka_enabled|bool %}{{ccloud_kafka_key}}{% else %}ksql{% endif %}',
        'password': '{% if ccloud_kafka_enabled|bool %}{{ccloud_kafka_secret}}{% else %}{{ksql_secret}}{% endif %}'
      }{% endif %}{% if 'control_center' in groups %},
      'control_center': {
        'principal': '{% if ccloud_kafka_enabled|bool %}{{ccloud_kafka_key}}{% else %}control_center{% endif %}',
        'password': '{% if ccloud_kafka_enabled|bool %}{{ccloud_kafka_secret}}{% else %}{{control_center_secret}}{% endif %}'
      }{% endif %}{% if 'kafka_connect_replicator' in groups %},
      'kafka_connect_replicator': {
        'principal': 'kafka_connect_replicator',
        'password': '{{kafka_connect_replicator_secret}}'
      }{% endif %}
    }"
    sasl_plain_users: {}
    zookeeper_digest_users:
      admin:
        principal: admin
        password: "{{admin_secret}}"
      kafka_broker:
        principal: kafka_broker
        password: "{{kafka_broker_secret}}"
    kafka_broker_rest_proxy_basic_users:
      admin:
        principal: admin
        password: "{{admin_secret}}"
        roles: admin
    _schema_registry_basic_users: "{
      'admin': {
        'principal': '{% if ccloud_schema_registry_enabled|bool %}{{ccloud_schema_registry_key}}{% else %}admin{% endif %}',
        'password': '{% if ccloud_schema_registry_enabled|bool %}{{ccloud_schema_registry_secret}}{% else %}{{admin_secret}}{% endif %}',
        'roles': 'admin'
      }
    }"
    kafka_connect_basic_users:
      admin:
        principal: admin
        password: "{{admin_secret}}"
    ksql_basic_users:
      admin:
        principal: admin
        password: "{{admin_secret}}"
        roles: admin
    kafka_rest_basic_users:
      admin:
        principal: admin
        password: "{{admin_secret}}"
        roles: admin
    control_center_basic_users:
      admin:
        principal: admin
        password: "{{admin_secret}}"
        roles: admin
zookeeper:
  hosts:
    kfk-1.domain.name:
    kfk-2.domain.name:
    kfk-3.domain.name:
kafka_broker:
  confluent.balancer.enable: "true"
  hosts:
    kfk-1.domain.name:
    kfk-2.domain.name:
    kfk-3.domain.name:
schema_registry:
  hosts:
    kfk-1.domain.name:
kafka_rest:
  hosts:
    kfk-1.domain.name:
ksql:
  hosts:
    kfk-2.domain.name:
kafka_connect:
  hosts:
    kfk-2.domain.name:
control_center:
  hosts:
    kfk-2.domain.name:

Logs Nothing failed, just incomplete run. Probably, problem is with the facts collection regarding packages deployed and needed to be updated.

Environment (please complete the following information):

• OS: Ubuntu 18.04
• CP-Ansible Branch: 6.2.2-post
• Ansible Version 2.9.2

Additional context just need help with fixing that behavior.

[nsharma-git]

Hello @ay-b For certs update, you should be running the all.yml playbook with --tags ssl Let me know how it goes. Thanks!

[louafi1510]

I am using kafka confluent version 7.0.1 and wanted to renew the keystores that are used by the brokers but when I execute the all.yml in combination with the --tags ssl "nothing" happens" (Perhaps because the all.yml has been changed with the current version of confluent?)

This is the output that I get when i run the all.yml with --tags ssl: ansible-playbook -i inventories/clustername/cp-clustername.yml -i inventories/clustername/afkl-clustername.yml /tech/kafka/ansible/kafkaconfluent/cp-ansible/all.yml --vault-id /tech/kafka/ansible/files/passwordfile.txt --user ansible --become -K --tags ssl [DEPRECATION WARNING]: DEFAULT_HASH_BEHAVIOUR option, This feature is fragile and not portable, leading to continual confusion and misuse , use the combine filter explicitly instead. This feature will be removed in version 2.13. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. BECOME password: [WARNING]: Could not match supplied host pattern, ignoring: kafka_connect_replicator

PLAY [Host Prerequisites] ************************************************************************************************************************************************************

TASK [confluent.common : Gather OS Facts] ******************************************************************************************************************************************** Thursday 23 March 2023 12:02:10 +0100 (0:00:00.067) 0:00:00.067 ******** ok: [zookeeperhost1] ok: [zookeeperhost2] ok: [brokerhost1] ok: [zookeeperhost3] ok: [brokerhost2] ok: [rangerhost]

PLAY [Zookeeper Status Finding] ****************************************************************************************************************************************************** [WARNING]: Could not match supplied host pattern, ignoring: zookeeper_parallel

PLAY [Zookeeper Parallel Provisioning] *********************************************************************************************************************************************** skipping: no hosts matched [WARNING]: Could not match supplied host pattern, ignoring: zookeeper_serial

PLAY [Zookeeper Serial Ordering] ***************************************************************************************************************************************************** skipping: no hosts matched [WARNING]: Could not match supplied host pattern, ignoring: zookeeper_follower

PLAY [Zookeeper Followers Provisioning] ********************************************************************************************************************************************** skipping: no hosts matched [WARNING]: Could not match supplied host pattern, ignoring: zookeeper_leader

PLAY [Zookeeper Leader Provisioning] ************************************************************************************************************************************************* skipping: no hosts matched

PLAY [Kafka Broker Status Finding] *************************************************************************************************************************************************** [WARNING]: Could not match supplied host pattern, ignoring: kafka_broker_parallel

PLAY [Kafka Broker Parallel Provisioning] ******************************************************************************************************************************************** skipping: no hosts matched [WARNING]: Could not match supplied host pattern, ignoring: kafka_broker_serial

PLAY [Kafka Broker Serial Ordering] ************************************************************************************************************************************************** skipping: no hosts matched [WARNING]: Could not match supplied host pattern, ignoring: kafka_broker_non_controller

PLAY [Kafka Broker Non Controllers Provisioning] ************************************************************************************************************************************* skipping: no hosts matched [WARNING]: Could not match supplied host pattern, ignoring: kafka_broker_controller

PLAY [Kafka Broker Controller Provisioning] ****************************************************************************************************************************************** skipping: no hosts matched

PLAY [Schema Registry Provisioning] ************************************************************************************************************************************************** skipping: no hosts matched

PLAY [Kafka Connect Status Finding] ************************************************************************************************************************************************** skipping: no hosts matched [WARNING]: Could not match supplied host pattern, ignoring: kafka_connect_parallel

PLAY [Kafka Connect Parallel Provisioning] ******************************************************************************************************************************************* skipping: no hosts matched [WARNING]: Could not match supplied host pattern, ignoring: kafka_connect_serial

PLAY [Kafka Connect Serial Provisioning] ********************************************************************************************************************************************* skipping: no hosts matched

PLAY [KSQL Status Finding] *********************************************************************************************************************************************************** skipping: no hosts matched [WARNING]: Could not match supplied host pattern, ignoring: ksql_parallel

PLAY [KSQL Parallel Provisioning] **************************************************************************************************************************************************** skipping: no hosts matched [WARNING]: Could not match supplied host pattern, ignoring: ksql_serial

PLAY [KSQL Serial Provisioning] ****************************************************************************************************************************************************** skipping: no hosts matched

PLAY [Kafka Rest Status Finding] ***************************************************************************************************************************************************** [WARNING]: Could not match supplied host pattern, ignoring: kafka_rest_parallel

PLAY [Kafka Rest Parallel Provisioning] ********************************************************************************************************************************************** skipping: no hosts matched [WARNING]: Could not match supplied host pattern, ignoring: kafka_rest_serial

PLAY [Kafka Rest Serial Provisioning] ************************************************************************************************************************************************ skipping: no hosts matched

PLAY [Control Center Status Finding] ************************************************************************************************************************************************* skipping: no hosts matched [WARNING]: Could not match supplied host pattern, ignoring: control_center_parallel

PLAY [Control Center Parallel Provisioning] ****************************************************************************************************************************************** skipping: no hosts matched [WARNING]: Could not match supplied host pattern, ignoring: control_center_serial

PLAY [Control Center Serial Provisioning] ******************************************************************************************************************************************** skipping: no hosts matched

PLAY [Kafka Connect Replicator Status Finding] *************************************************************************************************************************************** skipping: no hosts matched [WARNING]: Could not match supplied host pattern, ignoring: kafka_connect_replicator_parallel

PLAY [Kafka Connect Replicator Parallel Provisioning] ******************************************************************************************************************************** skipping: no hosts matched [WARNING]: Could not match supplied host pattern, ignoring: kafka_connect_replicator_serial

PLAY [Kafka Connect Replicator Serial Provisioning] ********************************************************************************************************************************** skipping: no hosts matched

PLAY RECAP *************************************************************************************************************************************************************************** zookeeperhost3 : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 zookeeperhost1 : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 zookeeperhost2 : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 rangerhost : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 brokerhost1 : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 brokerhost2 : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

when I list all the tags in all.yml i get the following output: ansible-playbook -i inventories/clustername/cp-clustername.yml -i inventories/clustername/afkl-clustername.yml /tech/kafka/ansible/kafkaconfluent/cp-ansible/all.yml --vault-id /tech/kafka/ansible/files/passwordfile.txt --user ansible --become -K --list-tags [DEPRECATION WARNING]: DEFAULT_HASH_BEHAVIOUR option, This feature is fragile and not portable, leading to continual confusion and misuse , use the combine filter explicitly instead. This feature will be removed in version 2.13. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. [WARNING]: Could not match supplied host pattern, ignoring: kafka_connect_replicator [WARNING]: Could not match supplied host pattern, ignoring: zookeeper_parallel [WARNING]: Could not match supplied host pattern, ignoring: zookeeper_serial [WARNING]: Could not match supplied host pattern, ignoring: zookeeper_follower [WARNING]: Could not match supplied host pattern, ignoring: zookeeper_leader [WARNING]: Could not match supplied host pattern, ignoring: kafka_broker_parallel [WARNING]: Could not match supplied host pattern, ignoring: kafka_broker_serial [WARNING]: Could not match supplied host pattern, ignoring: kafka_broker_non_controller [WARNING]: Could not match supplied host pattern, ignoring: kafka_broker_controller [WARNING]: Could not match supplied host pattern, ignoring: kafka_connect_parallel [WARNING]: Could not match supplied host pattern, ignoring: kafka_connect_serial [WARNING]: Could not match supplied host pattern, ignoring: ksql_parallel [WARNING]: Could not match supplied host pattern, ignoring: ksql_serial [WARNING]: Could not match supplied host pattern, ignoring: kafka_rest_parallel [WARNING]: Could not match supplied host pattern, ignoring: kafka_rest_serial [WARNING]: Could not match supplied host pattern, ignoring: control_center_parallel [WARNING]: Could not match supplied host pattern, ignoring: control_center_serial [WARNING]: Could not match supplied host pattern, ignoring: kafka_connect_replicator_parallel [WARNING]: Could not match supplied host pattern, ignoring: kafka_connect_replicator_serial

playbook: /tech/kafka/ansible/kafkaconfluent/cp-ansible/all.yml

play #1 (zookeeper:kafka_broker:schema_registry:kafka_connect:ksql:control_center:kafka_rest:kafka_connect_replicator): Host Prerequisites TAGS: [] TASK TAGS: [always, certificate_authority, common, masterkey, validate]

play #2 (zookeeper): Zookeeper Status Finding TAGS: [zookeeper] TASK TAGS: [zookeeper]

play #3 (zookeeper_parallel): Zookeeper Parallel Provisioning TAGS: [zookeeper] TASK TAGS: [common, health_check, package, ssl, zookeeper]

play #4 (zookeeper_serial): Zookeeper Serial Ordering TAGS: [zookeeper] TASK TAGS: [zookeeper]

play #5 (zookeeper_follower): Zookeeper Followers Provisioning TAGS: [zookeeper] TASK TAGS: [common, health_check, package, ssl, zookeeper]

play #6 (zookeeper_leader): Zookeeper Leader Provisioning TAGS: [zookeeper] TASK TAGS: [common, health_check, package, ssl, zookeeper]

play #7 (kafka_broker): Kafka Broker Status Finding TAGS: [kafka_broker] TASK TAGS: [kafka_broker]

play #8 (kafka_broker_parallel): Kafka Broker Parallel Provisioning TAGS: [kafka_broker] TASK TAGS: [common, health_check, kafka_broker, package, ssl, sysctl, systemd]

play #9 (kafka_broker_serial): Kafka Broker Serial Ordering TAGS: [kafka_broker] TASK TAGS: [kafka_broker]

play #10 (kafka_broker_non_controller): Kafka Broker Non Controllers Provisioning TAGS: [kafka_broker] TASK TAGS: [common, health_check, kafka_broker, package, ssl, sysctl, systemd]

play #11 (kafka_broker_controller): Kafka Broker Controller Provisioning TAGS: [kafka_broker] TASK TAGS: [common, health_check, kafka_broker, package, ssl, sysctl, systemd]

play #12 (schema_registry): Schema Registry Provisioning TAGS: [schema_registry] TASK TAGS: [common, health_check, package, schema_registry, ssl]

play #13 (kafka_connect): Kafka Connect Status Finding TAGS: [kafka_connect] TASK TAGS: [kafka_connect]

play #14 (kafka_connect_parallel): Kafka Connect Parallel Provisioning TAGS: [kafka_connect] TASK TAGS: [common, health_check, kafka_connect, package, ssl]

play #15 (kafka_connect_serial): Kafka Connect Serial Provisioning TAGS: [kafka_connect] TASK TAGS: [common, health_check, kafka_connect, package, ssl]

play #16 (ksql): KSQL Status Finding TAGS: [ksql] TASK TAGS: [ksql]

play #17 (ksql_parallel): KSQL Parallel Provisioning TAGS: [ksql] TASK TAGS: [common, health_check, ksql, package, ssl]

play #18 (ksql_serial): KSQL Serial Provisioning TAGS: [ksql] TASK TAGS: [common, health_check, ksql, package, ssl]

play #19 (kafka_rest): Kafka Rest Status Finding TAGS: [kafka_rest] TASK TAGS: [kafka_rest]

play #20 (kafka_rest_parallel): Kafka Rest Parallel Provisioning TAGS: [kafka_rest] TASK TAGS: [common, health_check, kafka_rest, package, ssl]

play #21 (kafka_rest_serial): Kafka Rest Serial Provisioning TAGS: [kafka_rest] TASK TAGS: [common, health_check, kafka_rest, package, ssl]

play #22 (control_center): Control Center Status Finding TAGS: [control_center] TASK TAGS: [control_center]

play #23 (control_center_parallel): Control Center Parallel Provisioning TAGS: [control_center] TASK TAGS: [common, control_center, health_check, package, ssl]

play #24 (control_center_serial): Control Center Serial Provisioning TAGS: [control_center] TASK TAGS: [common, control_center, health_check, package, ssl]

play #25 (kafka_connect_replicator): Kafka Connect Replicator Status Finding TAGS: [kafka_connect_replicator] TASK TAGS: [kafka_connect_replicator]

play #26 (kafka_connect_replicator_parallel): Kafka Connect Replicator Parallel Provisioning TAGS: [kafka_connect_replicator] TASK TAGS: [common, kafka_connect_replicator, package]

play #27 (kafka_connect_replicator_serial): Kafka Connect Replicator Serial Provisioning TAGS: [kafka_connect_replicator] TASK TAGS: [common, kafka_connect_replicator, package]

[gwbischof]

--tags ssl doesn't do anything for me also.