Enhancement to allow specify subscription channel for Kafka

[rene-oromtz]

Feature and motivation

This is an enhancement to work alongside IBM Storage Fusion. Storage Fusion BR requires AMQ-Streams, which if customer are using is the only operator allowed to use for Kafka, however, Fusion versions does not use "latest" operator version, in this case, for example, Fusion 2.9.0 (latest available) uses AMQ-Streams (now Streams for Apache Kafka) version 2.7.0-5 while the Kafka role attempts to install 2.8.0-0.

The following changes to Kafka role are suggested:

ansible-devops/roles/kafka/defaults/main.yaml From:

# vars for red hat amq or strimzi
amq_exists: false # initialize variable
kafka_defaults:
  strimzi:
    version: "3.7.0" # Supported versions for strimzi-cluster-operator.v0.40.0 are: [3.6.0, 3.6.1, 3.7.0]
    namespace: "strimzi"
    operator_name: "strimzi-kafka-operator"
    alias_name: "Strimzi"
  redhat:
    version: "3.7.0"
    namespace: "amq-streams"
    operator_name: "amq-streams"
    alias_name: "Red Hat AMQ Streams"
  aws:
    version: "3.3.1"

kafka_version: "{{ lookup('env', 'KAFKA_VERSION') | default(kafka_defaults[kafka_provider].version, true) }}"
kafka_namespace: "{{ lookup('env', 'KAFKA_NAMESPACE') | default(kafka_defaults[kafka_provider].namespace, true) }}"
kafka_operator_name: "{{ kafka_defaults[kafka_provider].operator_name }}"

To:

# vars for red hat amq or strimzi
amq_exists: false # initialize variable
kafka_defaults:
  strimzi:
    channel: "stable"
    version: "3.7.0" # Supported versions for strimzi-cluster-operator.v0.40.0 are: [3.6.0, 3.6.1, 3.7.0]
    namespace: "strimzi"
    operator_name: "strimzi-kafka-operator"
    alias_name: "Strimzi"
  redhat:
    channel: "stable"
    version: "3.7.0"
    namespace: "amq-streams"
    operator_name: "amq-streams"
    alias_name: "Red Hat AMQ Streams"
  aws:
    channel: "stable"
    version: "3.3.1"

kafka_channel: "{{ lookup('env', 'KAFKA_CHANNEL') | default(kafka_defaults[kafka_provider].channel, true) }}"
kafka_version: "{{ lookup('env', 'KAFKA_VERSION') | default(kafka_defaults[kafka_provider].version, true) }}"
kafka_namespace: "{{ lookup('env', 'KAFKA_NAMESPACE') | default(kafka_defaults[kafka_provider].namespace, true) }}"
kafka_operator_name: "{{ kafka_defaults[kafka_provider].operator_name }}"

ansible-devops/roles/kafka/tasks/provider/redhat/install-kafka.yml From:

# 2. Lookup the packagemanifest for amq-streams / strimzi
# -----------------------------------------------------------------------------
- name: Get {{ kafka_operator_name }} package manifest
  kubernetes.core.k8s_info:
    api_version: packages.operators.coreos.com/v1
    kind: PackageManifest
    name: "{{ kafka_operator_name }}"
    namespace: openshift-marketplace # Note: A namespace must be provided when calling packages.operators.coreos.com/v1
  register: kafka_manifest

- name: Assert that PackageManifest exists
  ansible.builtin.assert:
    that:
      - kafka_manifest is defined
      - kafka_manifest.resources is defined
      - kafka_manifest.resources | length == 1
    fail_msg: "PackageManifest not found: {{ kafka_operator_name }}"

- name: Set the subscription information
  set_fact:
    kafka_source: "{{ kafka_manifest.resources[0].status.catalogSource }}"
    kafka_source_namespace: "{{ kafka_manifest.resources[0].status.catalogSourceNamespace }}"
    kafka_default_channel: "{{ kafka_manifest.resources[0].status.defaultChannel }}"

To:

# 2. Lookup the packagemanifest for amq-streams / strimzi
# -----------------------------------------------------------------------------
- name: Get {{ kafka_operator_name }} package manifest
  kubernetes.core.k8s_info:
    api_version: packages.operators.coreos.com/v1
    kind: PackageManifest
    name: "{{ kafka_operator_name }}"
    namespace: openshift-marketplace # Note: A namespace must be provided when calling packages.operators.coreos.com/v1
  register: kafka_manifest

- name: Assert that PackageManifest exists
  ansible.builtin.assert:
    that:
      - kafka_manifest is defined
      - kafka_manifest.resources is defined
      - kafka_manifest.resources | length == 1
    fail_msg: "PackageManifest not found: {{ kafka_operator_name }}"

- name: Set the subscription information
  set_fact:
    kafka_source: "{{ kafka_manifest.resources[0].status.catalogSource }}"
    kafka_source_namespace: "{{ kafka_manifest.resources[0].status.catalogSourceNamespace }}"
    kafka_default_channel: "{{ kafka_channel | default(kafka_manifest.resources[0].status.defaultChannel) }}"

Usage example

In order to avoid having conflicting versions of the operator, this enhancement can be made to allow user to specify default subscription channel, in this case, to match as closely as possible with Fusion, we can choose channel amq-streams-2.7.x instead of stable, the installed version will be 2.7.0-7 which is not the same as the one Fusion uses but at least is not a bigger "jump" between versions as is just a minor release.

This can also provide more control over the Kafka version the customer want to use.