Allow to set static PVC name for a broker

[AndrewBoklashko]

Hi, thanks for the great Kafka operator. We've been using it on our dev environments for quite a long time and it's working well for us.

My new use case is to setup Kafka cluster in on-premise Kubernetes cluster without dynamic persistent volumes provisioning. So I am using preexisting local volumes which I want to pre-bind to Kafka brokers PVCs using claimRef on PVs. This way volumes will always be dedicated to Kafka brokers even if cluster needs to be rebuilt.

The problem is that operator always creates PVC with generated name. Considering that, can we allow to set PVC metadata, including name, in StorageConfigs? Note, I understand that I can set volumeName on PVC, however it will require me to release the volume manually if I need to rebuild Kafka cluster.

[baluchicken]

Hi @AndrewBoklashko sorry for the delayed response. We are considering your proposal to allow setting PVC metadata. In the mean time there is a workaround to your problem. The operator using the client list method to find the required PVCs. If these PVCs are not present it will create new ones, but it can be tricked to reuse already created ones. To do that all you have to do is add specific labels and annotations to your PVC.

  annotations:
    mountPath: "<your mount path specified in your kafkacluster cr spec>"
  labels:
    brokerId: "<brokerId>"
    app: kafka
    kafka_cr: <your kafka cr name>

For example in case of the following CR:

apiVersion: kafka.banzaicloud.io/v1beta1
kind: KafkaCluster
metadata:
  name: kafka
spec:
  brokerConfigGroups:
    default:
      storageConfigs:
        - mountPath: "/kafka-logs"
          pvcSpec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 10Gi
      brokers:
        - id: 0
           brokerConfigGroup: "default"

The following PVC will be reused by the operator without placing any ownerreference:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: kafka-0
  namespace: kafka
  annotations:
    mountPath: "/kafka-logs"
  labels:
    brokerId: "0"
    app: kafka
    kafka_cr: kafka
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

[AndrewBoklashko]

@baluchicken thanks, I will try the provided workaround.

[stoader]

@AndrewBoklashko support for pre-provisioned volumes has been added to koperator(see https://banzaicloud.com/docs/supertubes/kafka-operator/pre-provisioned-volumes/). This feature is currently available on master branch and will be included into the next release.

cc @pregnor @hi-im-aren