How to start a single node kafka cluster

[tobias-neubert]

Hi,

can anybody tell me how to start only one zookeeper and one kafka broker? Given the following values.yaml file in my helm chart I can enable/disable all components but the number of brokers/servers is not adjusted accordingly. What am I doing wrong here?

cp-zookeeper:
  enabled: true
  servers: 1
cp-kafka:
  enabled: true
  brokers: 1
cp-schema-registry:
  enabled: false
cp-kafka-connect:
  enabled: false
cp-kafka-rest:
  enabled: false
cp-ksql-server:
  enabled: false

[jswoods]

@tobias-neubert I think this is what you want. I also found the namespacing confusing, but I think this is because the logic for the enabled boolean is done as a conditional around including the sub charts.

cp-helm-charts:
  cp-zookeeper:
    servers: 1
  cp-kafka:
    brokers: 1
cp-zookeeper:
  enabled: true
cp-kafka:
  enabled: true
cp-schema-registry:
  enabled: false
cp-kafka-connect:
  enabled: false
cp-kafka-rest:
  enabled: false
cp-ksql-server:
  enabled: false

[sangloe]

Is there a comprehensive list of values that need to be updated to run a single zk/kafka cluster? I've set all the replication factor settings I can find to "1" but there still seems to be an internal segment.ms topic that tries to be created on startup and fails with the error that the replication factor (3) is higher than the broker count (1). It would be nice to use these charts for local development on Docker desktop. They work fine as-is but it would be nice to get some memory back by only running a single node.

[Shabahang]

I had same issue. following solved my problem:

1- I added some parameters on values.yaml and instelled kafka.

cp-kafka:
  brokers: 1
  configurationOverrides:
    "offsets.topic.replication.factor": "1"
    "default.replication.factor": 1
    "min.insync.replicas": 1
    "confluent.metrics.reporter.topic.replicas": 1
    "transaction.state.log.replication.factor": 1
    "confluent.tier.metadata.replication.factor": 1
    "confluent.balancer.topic.replication.factor": 1
    "confluent.durability.topic.replication.factor": 1

cp-kafka-connect:
  configurationOverrides:
    "config.storage.replication.factor": "1"
    "offset.storage.replication.factor": "1"
    "status.storage.replication.factor": "1"


cp-control-center:
  configurationOverrides:
    "internal.topics.replication": "1"
    "monitoring.interceptor.topic.replication": "1"
    "metrics.topic.replication": "1"
    "command.topic.replication": "1"
    "replication.factor": "1"


cp-schema-registry:
  configurationOverrides:
    "kafkastore.topic.replication.factor" : 1

2- Then created the following 3 topics manually, (num_partitions=12, replication_factor=1):

• '_confluent-telemetry-metrics'
• '_confluent-license'
• '_confluent_balancer_broker_samples'

Functionality of the system is not yet masively tested. But there is no more restart and looks that system is stable.

[vjvel]

@Shabahang I configured the parameters which you suggested. i can create new topics but below error is persist.

[2022-06-30 15:39:32,046] ERROR [KafkaApi-0] Number of alive brokers '1' does not meet the required replication factor '3' for the offsets topic (configured via 'offsets.topic.replication.factor'). This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis)
[2022-06-30 15:39:32,062] ERROR [KafkaApi-0] Number of alive brokers '1' does not meet the required replication factor '3' for the offsets topic (configured via 'offsets.topic.replication.factor'). This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis)
[2022-06-30 15:39:32,148] ERROR [KafkaApi-0] Number of alive brokers '1' does not meet the required replication factor '3' for the offsets topic (configured via 'offsets.topic.replication.factor'). This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis)
[2022-06-30 15:39:32,162] ERROR [KafkaApi-0] Number of alive brokers '1' does not meet the required replication factor '3' for the offsets topic (configured via 'offsets.topic.replication.factor'). This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis)

Can anyone help me on this ? how to get rid of this error ?

[vjvel]

@tobias-neubert did you manage to fix this issue in single broker ?

[Shabahang]

@vjvel @tobias-neubert The problem is that you did not recreate topics with replicatin_factor 1 . Please follow the steps again :

1- Use the values.yaml files to install. 2- 3 topics should be dropped and recreated with some parameters for single broker. So get a bash on the broker container and run followings :

/bin/kafka-topics --bootstrap-server localhost:9092 --delete --topic _confluent-telemetry-metrics /bin/kafka-topics --bootstrap-server localhost:9092 --create --topic _confluent-telemetry-metrics --replication-factor 1 --partitions 12

/bin/kafka-topics --bootstrap-server localhost:9092 --delete --topic _confluent-license /bin/kafka-topics --bootstrap-server localhost:9092 --create --topic _confluent-license --replication-factor 1 --partitions 12

/bin/kafka-topics --bootstrap-server localhost:9092 --delete --topic _confluent_balancer_broker_samples /bin/kafka-topics --bootstrap-server localhost:9092 --create --topic _confluent_balancer_broker_samples --replication-factor 1 --partitions 12

(OPTIONAL): 3- if you need to minimize the retention of metrics run following too :

/bin/kafka-configs --bootstrap-server localhost:9092 --alter --topic _confluent-metrics --add-config retention.ms=10800000 /bin/kafka-configs --bootstrap-server localhost:9092 --alter --topic _confluent-controlcenter-7-0-0-1-monitoring-message-rekey-store --add-config retention.ms=3600000 /bin/kafka-configs --bootstrap-server localhost:9092 --alter --topic _confluent-controlcenter-7-0-0-1-monitoring-trigger-event-rekey --add-config retention.ms=3600000 /bin/kafka-configs --bootstrap-server localhost:9092 --alter --topic _confluent-controlcenter-7-0-0-1-metrics-trigger-measurement-rekey --add-config retention.ms=3600000 /bin/kafka-configs --bootstrap-server localhost:9092 --alter --topic _confluent-controlcenter-7-0-0-1-MetricsAggregateStore-repartition --add-config retention.ms=3600000 /bin/kafka-configs --bootstrap-server localhost:9092 --alter --topic _confluent-controlcenter-7-0-0-1-MetricsAggregateStore-changelog --add-config retention.ms=28800000