Multiple kafka clusters not working

[Fattouh92]

Describe the bug Following the steps described here https://cloudflow.io/docs/current/administration/installing-cloudflow.html#_default_and_named_kafka_clusters.

We have the following values that we pass to helm install

kafkaClusters:
  default:
    bootstrapServers: xxx.cloudflow:9092
    partitions: 53
    replicas: 3
    connectionConfig: {}
    producerConfig: {}
    consumerConfig: {}
  another-one:
    bootstrapServers: yyy.cloudflow:9092
    partitions: 53
    replicas: 3
    connectionConfig: {}
    producerConfig: {}
    consumerConfig: {}

the result is two secrets created in the cloudflow namespace that have the label cloudflow.lightbend.com/kafka-cluster-name set accordingly.

In the blueprint we have the following:

blueprint {
  streamlets {
    ingress = Ingress
    processor = Processor
  }

  topics {
    processing-topic {
      producers = [ingress.out]
      consumers = [processor.in]
      cluster = another-one
    }
  }
}

this results in the error

error [kubectl-cloudflow] Failure
akka.cli.cloudflow.CliException: Could not find the kafka cluster configurations: [another-one] referenced in the Cr
        at akka.cli.cloudflow.CliException$.apply(CliException.scala:9)
        at akka.cli.cloudflow.execution.DeployExecution.$anonfun$referencedKafkaSecretExists$3(DeployExecution.scala:128)
        at scala.util.Success.map(Try.scala:262)
        at akka.cli.cloudflow.execution.DeployExecution.referencedKafkaSecretExists(DeployExecution.scala:121)
        at akka.cli.cloudflow.execution.DeployExecution.$anonfun$run$23(DeployExecution.scala:193)
        at scala.util.Success.flatMap(Try.scala:258)
        at akka.cli.cloudflow.execution.DeployExecution.$anonfun$run$22(DeployExecution.scala:188)
        at scala.util.Success.flatMap(Try.scala:258)
        at akka.cli.cloudflow.execution.DeployExecution.$anonfun$run$17(DeployExecution.scala:182)
        at scala.util.Success.flatMap(Try.scala:258)
        at scala.util.Try$WithFilter.flatMap(Try.scala:139)
        at akka.cli.cloudflow.execution.DeployExecution.$anonfun$run$13(DeployExecution.scala:175)
        at scala.util.Success.flatMap(Try.scala:258)
        at akka.cli.cloudflow.execution.DeployExecution.$anonfun$run$11(DeployExecution.scala:171)
        at scala.util.Success.flatMap(Try.scala:258)
        at akka.cli.cloudflow.execution.DeployExecution.$anonfun$run$9(DeployExecution.scala:166)
        at scala.util.Success.flatMap(Try.scala:258)
        at akka.cli.cloudflow.execution.DeployExecution.$anonfun$run$8(DeployExecution.scala:165)
        at scala.util.Success.flatMap(Try.scala:258)
        at akka.cli.cloudflow.execution.DeployExecution.$anonfun$run$7(DeployExecution.scala:164)
        at scala.util.Success.flatMap(Try.scala:258)
        at akka.cli.cloudflow.execution.DeployExecution.$anonfun$run$5(DeployExecution.scala:162)
        at scala.util.Success.flatMap(Try.scala:258)
        at akka.cli.cloudflow.execution.DeployExecution.$anonfun$run$2(DeployExecution.scala:158)
        at scala.util.Success.flatMap(Try.scala:258)
        at akka.cli.cloudflow.execution.DeployExecution.run(DeployExecution.scala:149)
        at akka.cli.cloudflow.Cli.run(Cli.scala:39)
        at akka.cli.cloudflow.Main$.run(Main.scala:32)
        at akka.cli.cloudflow.Main$.main(Main.scala:18)
Error: Could not find the kafka cluster configurations: [another-one] referenced in the Cr

Removing the cluster line in the blueprint, the cli uses the default cluster and it deploys normally

To Reproduce refer to the blueprint, helm install values supplied in the previous section

Expected behavior the cli deploys the app normally and uses the refrenced kafka cluster in the blueprint

Screenshots If applicable, add screenshots to help explain your problem.

Additional context Add any other context about the problem here.

[thomasschoeftner]

Are there any update on this one? We are seeing the same issue when using multiple brokers.

Especially when sourcing Kafka externally (e.g. as SaaS) this will get messy, IMO. Many thanks!

[RayRoestenburg]

Can you provide a reproducer? I can't see where processing-poc comes from.

[RayRoestenburg]

Let me see if this happens by simply using the feature..

[Fattouh92]

@RayRoestenburg yeah my bad. the error includes the cluster name used in the blueprint file topics section. so in this case another-one cluster is in the error since it is the referenced cluster in the blueprint

I updated my initial comment to be complete.

I will provide a producer when my time allows for it. hopefully max by end of week

thanks for your support