camel-kafka-connector
camel-kafka-connector copied to clipboard
Published
20 hours ago •
apache
apache
Camel KafkaConnect pods can't start to run
The pods created by the KafkaConnect can't start, when I followed the document to create a KafkaConnect to backup/sink the Kafka data to aws s3.
I'm wondering if there other external steps that need to be added to the guideline.
Troubleshooting
- create the
kafkaconnect
cat <<EOF | oc apply -f -
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
name: kafka-connect-cluster
annotations:
# enable connect operator
strimzi.io/use-connector-resources: "true"
spec:
image: $REGISTRY_URL/$REGISTRY_USR/camel-kafka-connector-plugin:latest
version: 2.3.0
replicas: 1
bootstrapServers: kafka-kafka-bootstrap.multicluster-global-hub.svc:9092
externalConfiguration:
env:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: aws-creds
key: awsAccessKey
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: aws-creds
key: awsSecretAccessKey
resources:
requests:
cpu: 250m
memory: 512Mi
limits:
cpu: 500m
memory: 1Gi
jvmOptions:
gcLoggingEnabled: false
config:
group.id: kafka-connect-cluster
key.converter: org.apache.kafka.connect.storage.StringConverter
value.converter: org.apache.kafka.connect.storage.StringConverter
offset.storage.topic: kafka-connect-cluster-offsets
config.storage.topic: kafka-connect-cluster-configs
status.storage.topic: kafka-connect-cluster-status
config.storage.replication.factor: 1
offset.storage.replication.factor: 1
status.storage.replication.factor: 1
EOF
kafkaconnect.kafka.strimzi.io/kafka-connect-cluster created
- get pod
$ oc get pods
NAME READY STATUS RESTARTS AGE
kafka-connect-cluster-connect-58cd465cc7-qsgng 0/1 Running 2 (29s ago) 55s
- describe pod
$ oc describe pod kafka-connect-cluster-connect-58cd465cc7-qsgng
Name: kafka-connect-cluster-connect-58cd465cc7-qsgng
Namespace: multicluster-global-hub
Priority: 0
Node: ip-10-0-132-6.ec2.internal/10.0.132.6
Start Time: Mon, 29 Jan 2024 12:59:03 +0000
Labels: app.kubernetes.io/instance=kafka-connect-cluster
app.kubernetes.io/managed-by=strimzi-cluster-operator
app.kubernetes.io/name=kafka-connect
app.kubernetes.io/part-of=strimzi-kafka-connect-cluster
pod-template-hash=58cd465cc7
strimzi.io/cluster=kafka-connect-cluster
strimzi.io/component-type=kafka-connect
strimzi.io/kind=KafkaConnect
strimzi.io/name=kafka-connect-cluster-connect
Annotations: k8s.v1.cni.cncf.io/network-status:
[{
"name": "openshift-sdn",
"interface": "eth0",
"ips": [
"10.128.1.50"
],
"default": true,
"dns": {}
}]
k8s.v1.cni.cncf.io/networks-status:
[{
"name": "openshift-sdn",
"interface": "eth0",
"ips": [
"10.128.1.50"
],
"default": true,
"dns": {}
}]
openshift.io/scc: restricted-v2
seccomp.security.alpha.kubernetes.io/pod: runtime/default
strimzi.io/auth-hash: 0
strimzi.io/logging-appenders-hash: 06ee78c4
Status: Running
IP: 10.128.1.50
IPs:
IP: 10.128.1.50
Controlled By: ReplicaSet/kafka-connect-cluster-connect-58cd465cc7
Containers:
kafka-connect-cluster-connect:
Container ID: cri-o://e5cc2aadcfe9877521bda830254be72fcfaca434d467714458b66b4560ca13c8
Image: quay.io/myan/camel-kafka-connector-plugin:latest
Image ID: quay.io/myan/camel-kafka-connector-plugin@sha256:bfb98f6f879b216f9d65634a1d2da25b405fd8f5ad17e7c53bb8e3b6a1236237
Port: 8083/TCP
Host Port: 0/TCP
Args:
/opt/kafka/kafka_connect_run.sh
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Mon, 29 Jan 2024 12:59:48 +0000
Finished: Mon, 29 Jan 2024 12:59:59 +0000
Ready: False
Restart Count: 2
Liveness: http-get http://:rest-api/ delay=60s timeout=5s period=10s #success=1 #failure=3
Readiness: http-get http://:rest-api/ delay=60s timeout=5s period=10s #success=1 #failure=3
Environment:
KAFKA_CONNECT_CONFIGURATION: offset.storage.topic=connect-cluster-offsets
value.converter=org.apache.kafka.connect.json.JsonConverter
config.storage.topic=connect-cluster-configs
key.converter=org.apache.kafka.connect.json.JsonConverter
group.id=connect-cluster
status.storage.topic=connect-cluster-status
KAFKA_CONNECT_METRICS_ENABLED: false
KAFKA_CONNECT_BOOTSTRAP_SERVERS: kafka-kafka-bootstrap.multicluster-global-hub.svc:9092
STRIMZI_KAFKA_GC_LOG_ENABLED: false
KAFKA_HEAP_OPTS: -Xms128M
AWS_ACCESS_KEY_ID: <set to the key 'awsAccessKey' in secret 'aws-creds'> Optional: false
AWS_SECRET_ACCESS_KEY: <set to the key 'awsSecretAccessKey' in secret 'aws-creds'> Optional: false
Mounts:
/opt/kafka/custom-config/ from kafka-metrics-and-logging (rw)
/tmp from strimzi-tmp (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-srz7b (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
strimzi-tmp:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: 5Mi
kafka-metrics-and-logging:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: kafka-connect-cluster-connect-config
Optional: false
# Please edit the object below. Lines beginning with a '#' will be ignored,
kube-api-access-srz7b:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
ConfigMapName: openshift-service-ca.crt
ConfigMapOptional: <nil>
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 72s default-scheduler Successfully assigned multicluster-global-hub/kafka-connect-cluster-connect-58cd465cc7-qsgng to ip-10-0-132-6.ec2.internal
Normal AddedInterface 70s multus Add eth0 [10.128.1.50/23] from openshift-sdn
Normal Pulled 70s kubelet Successfully pulled image "quay.io/myan/camel-kafka-connector-plugin:latest" in 171.0038ms (171.01734ms including waiting)
Normal Pulled 57s kubelet Successfully pulled image "quay.io/myan/camel-kafka-connector-plugin:latest" in 160.465058ms (160.478918ms including waiting)
Normal Pulling 26s (x3 over 70s) kubelet Pulling image "quay.io/myan/camel-kafka-connector-plugin:latest"
Normal Created 26s (x3 over 69s) kubelet Created container kafka-connect-cluster-connect
Normal Started 26s (x3 over 69s) kubelet Started container kafka-connect-cluster-connect
Normal Pulled 26s kubelet Successfully pulled image "quay.io/myan/camel-kafka-connector-plugin:latest" in 171.659524ms (171.671444ms including waiting)
Warning BackOff 11s (x4 over 45s) kubelet Back-off restarting failed container
- pod log
$ oc logs -f kafka-connect-cluster-connect-58cd465cc7-qsgng
Preparing truststore
Preparing truststore is complete
Starting Kafka Connect with configuration:
# Bootstrap servers
bootstrap.servers=kafka-kafka-bootstrap.multicluster-global-hub.svc:9092
# REST Listeners
...
2024-01-29 13:47:33,292 INFO Kafka version: 2.6.0 (org.apache.kafka.common.utils.AppInfoParser) [DistributedHerder-connect-1-1]
2024-01-29 13:47:33,292 INFO Kafka commitId: 62abe01bee039651 (org.apache.kafka.common.utils.AppInfoParser) [DistributedHerder-connect-1-1]
2024-01-29 13:47:33,293 INFO Kafka startTimeMs: 1706536053292 (org.apache.kafka.common.utils.AppInfoParser) [DistributedHerder-connect-1-1]
2024-01-29 13:47:33,297 INFO [Consumer clientId=consumer-kafka-connect-cluster-1, groupId=kafka-connect-cluster] Cluster ID: IbXwdCfQRzqSPgS22-GhAg (org.apache.kafka.clients.Metadata) [DistributedHerder-connect-1-1]
2024-01-29 13:47:33,299 ERROR [Worker clientId=connect-1, groupId=kafka-connect-cluster] Uncaught exception in herder work thread, exiting: (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [DistributedHerder-connect-1-1]
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [kafka-connect-cluster-offsets]
2024-01-29 13:47:33,393 INFO Stopped http_8083@15d42ccb{HTTP/1.1,[http/1.1]}{0.0.0.0:8083} (org.eclipse.jetty.server.AbstractConnector) [Thread-3]
2024-01-29 13:47:33,403 INFO DefaultSessionIdManager workerName=node0 (org.eclipse.jetty.server.session) [main]
2024-01-29 13:47:33,403 INFO No SessionScavenger set, using defaults (org.eclipse.jetty.server.session) [main]
2024-01-29 13:47:33,404 INFO node0 Scavenging every 600000ms (org.eclipse.jetty.server.session) [main]
2024-01-29 13:47:35,090 INFO Started o.e.j.s.ServletContextHandler@22bf9122{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler) [main]
2024-01-29 13:47:35,090 INFO REST resources initialized; server is started and ready to handle requests (org.apache.kafka.connect.runtime.rest.RestServer) [main]
2024-01-29 13:47:35,090 INFO Kafka Connect started (org.apache.kafka.connect.runtime.Connect) [main]
2024-01-29 13:47:35,090 INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect) [connect-shutdown-hook]
2024-01-29 13:47:35,090 INFO Stopping REST server (org.apache.kafka.connect.runtime.rest.RestServer) [connect-shutdown-hook]
2024-01-29 13:47:35,091 INFO REST server stopped (org.apache.kafka.connect.runtime.rest.RestServer) [connect-shutdown-hook]
2024-01-29 13:47:35,091 INFO [Worker clientId=connect-1, groupId=kafka-connect-cluster] Herder stopping (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [connect-shutdown-hook]
2024-01-29 13:47:40,091 INFO [Worker clientId=connect-1, groupId=kafka-connect-cluster] Herder stopped (org.apache.kafka.connect.runtime.distributed.DistributedHerder) [connect-shutdown-hook]
2024-01-29 13:47:40,091 INFO Kafka Connect stopped (org.apache.kafka.connect.runtime.Connect) [connect-shutdown-hook]
This is something related to how you configure strimzi. You're not authorized to access the topic.
